[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:13:33 dist-test-slave-3g67 krb5kdc[6426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:13:33 dist-test-slave-3g67 krb5kdc[6426](info): set up 2 sockets
May 03 08:13:33 dist-test-slave-3g67 krb5kdc[6426](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20240503 08:13:34.775930 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:13:34 dist-test-slave-3g67 krb5kdc[6426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:13:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:13:34.895347 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40851
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:34.976562 6439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:34.976805 6439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:34.976830 6439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:34.978289 6439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:34.978363 6439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:34.978377 6439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:34.978395 6439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:34.978407 6439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:34.980309 6439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:34.980578 6439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:34.980803 6439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:34.985870 6447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:34.986279 6445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:34.987258 6444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:34.988291 6439 server_base.cc:1034] running on GCE node
I20240503 08:13:34.988616 6439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:34.988926 6439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:34.990131 6439 hybrid_clock.cc:648] HybridClock initialized: now 1714724014990115 us; error 31 us; skew 500 ppm
May 03 08:13:34 dist-test-slave-3g67 krb5kdc[6426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724014, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:34.993005 6439 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:13:34.993463 6439 webserver.cc:460] Webserver started at http://127.6.68.254:36401/ using document root <none> and password file <none>
I20240503 08:13:34.993638 6439 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:34.993675 6439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:34.993767 6439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:34.994794 6439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "79f0289f7d2643299e5f33d97888bae6"
format_stamp: "Formatted at 2024-05-03 08:13:34 on dist-test-slave-3g67"
server_key: "150020121cab1ed8fb14874d20be4044"
server_key_iv: "190245787d827530e8511a4faae8abed"
server_key_version: "encryptionkey@0"
I20240503 08:13:34.995081 6439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "79f0289f7d2643299e5f33d97888bae6"
format_stamp: "Formatted at 2024-05-03 08:13:34 on dist-test-slave-3g67"
server_key: "150020121cab1ed8fb14874d20be4044"
server_key_iv: "190245787d827530e8511a4faae8abed"
server_key_version: "encryptionkey@0"
I20240503 08:13:34.996449 6439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:13:34.997210 6454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:34.997437 6439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:34.997519 6439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "79f0289f7d2643299e5f33d97888bae6"
format_stamp: "Formatted at 2024-05-03 08:13:34 on dist-test-slave-3g67"
server_key: "150020121cab1ed8fb14874d20be4044"
server_key_iv: "190245787d827530e8511a4faae8abed"
server_key_version: "encryptionkey@0"
I20240503 08:13:34.997588 6439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:35.016445 6439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:35.018770 6439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:35.018887 6439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:35.022598 6439 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:40851
I20240503 08:13:35.022656 6507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:40851 every 8 connection(s)
I20240503 08:13:35.022962 6439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:13:35.023981 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.026482 6508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6: Bootstrap starting.
I20240503 08:13:35.027102 6508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:35.027384 6508 log.cc:824] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:35.028156 6508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6: No bootstrap required, opened a new log
I20240503 08:13:35.029727 6508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0289f7d2643299e5f33d97888bae6" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40851 } }
I20240503 08:13:35.029879 6508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:35.029911 6508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f0289f7d2643299e5f33d97888bae6, State: Initialized, Role: FOLLOWER
I20240503 08:13:35.030059 6508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0289f7d2643299e5f33d97888bae6" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40851 } }
I20240503 08:13:35.030164 6508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:35.030222 6508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:35.030274 6508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:35.030465 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 6439
I20240503 08:13:35.030550 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:13:35.031059 6508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0289f7d2643299e5f33d97888bae6" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40851 } }
I20240503 08:13:35.031116 6419 external_mini_cluster.cc:1428] Setting key 3f2a0a38368134f2d13ead670a946a6e
I20240503 08:13:35.031176 6508 leader_election.cc:302] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79f0289f7d2643299e5f33d97888bae6; no voters:
I20240503 08:13:35.031370 6508 leader_election.cc:288] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:35.031421 6512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:35.031687 6508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:35.032668 6512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [term 1 LEADER]: Becoming Leader. State: Replica: 79f0289f7d2643299e5f33d97888bae6, State: Running, Role: LEADER
I20240503 08:13:35.032868 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0289f7d2643299e5f33d97888bae6" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40851 } }
I20240503 08:13:35.033288 6516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79f0289f7d2643299e5f33d97888bae6. Latest consensus state: current_term: 1 leader_uuid: "79f0289f7d2643299e5f33d97888bae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0289f7d2643299e5f33d97888bae6" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40851 } } }
I20240503 08:13:35.033375 6516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:35.033524 6513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79f0289f7d2643299e5f33d97888bae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0289f7d2643299e5f33d97888bae6" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40851 } } }
I20240503 08:13:35.033627 6513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:35.033818 6525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:35.034555 6525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:35.036190 6525 catalog_manager.cc:1347] Generated new cluster ID: e5c03077a43546d9a4dcafd1e97e0dd6
I20240503 08:13:35.036252 6525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:35.043349 6525 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:35.044039 6525 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:35.047725 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:35.031976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.6.68.254:40851)
0503 08:13:35.032930 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:35.032944 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:35.032975 (+ 31us) server_negotiation.cc:407] Connection header received
0503 08:13:35.034583 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:35.034591 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:35.034645 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:35.034811 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:35.036723 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.037938 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:35.038812 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.039089 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:35.043312 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:35.043320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:35.043320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:35.043348 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:35.045769 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.046362 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.046365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.046365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.046432 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.046652 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.046653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.046653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.046875 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:35.047026 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:35.047280 (+ 254us) server_negotiation.cc:299] Negotiation successful
0503 08:13:35.047430 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":632,"threads_started":1}
I20240503 08:13:35.055819 6525 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 79f0289f7d2643299e5f33d97888bae6: Generated new TSK 0
I20240503 08:13:35.056046 6525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:13:35.087538 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40851
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:35.173859 6535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:35.174083 6535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:35.174110 6535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:35.175544 6535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:35.175604 6535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:35.175673 6535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:13:35.177398 6535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40851
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:35.177654 6535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:35.177870 6535 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:35.183471 6540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:35.184083 6541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:35.184996 6543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:35.185910 6535 server_base.cc:1034] running on GCE node
I20240503 08:13:35.186178 6535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:35.186465 6535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:35.187676 6535 hybrid_clock.cc:648] HybridClock initialized: now 1714724015187651 us; error 40 us; skew 500 ppm
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724015, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:35.190582 6535 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:13:35.191152 6535 webserver.cc:460] Webserver started at http://127.6.68.193:35115/ using document root <none> and password file <none>
I20240503 08:13:35.191404 6535 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:35.191463 6535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:35.191579 6535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:35.192719 6535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "8f67eefdb38c495e90b1063b97e3d79a"
format_stamp: "Formatted at 2024-05-03 08:13:35 on dist-test-slave-3g67"
server_key: "07dcbfa8db44a8207f18dbcf7ec78b41"
server_key_iv: "22a6eef72cb6ad104c71fa2452f3ad6d"
server_key_version: "encryptionkey@0"
I20240503 08:13:35.192993 6535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "8f67eefdb38c495e90b1063b97e3d79a"
format_stamp: "Formatted at 2024-05-03 08:13:35 on dist-test-slave-3g67"
server_key: "07dcbfa8db44a8207f18dbcf7ec78b41"
server_key_iv: "22a6eef72cb6ad104c71fa2452f3ad6d"
server_key_version: "encryptionkey@0"
I20240503 08:13:35.194339 6535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:13:35.195086 6550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:35.195279 6535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:35.195354 6535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "8f67eefdb38c495e90b1063b97e3d79a"
format_stamp: "Formatted at 2024-05-03 08:13:35 on dist-test-slave-3g67"
server_key: "07dcbfa8db44a8207f18dbcf7ec78b41"
server_key_iv: "22a6eef72cb6ad104c71fa2452f3ad6d"
server_key_version: "encryptionkey@0"
I20240503 08:13:35.195449 6535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:35.205421 6535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:35.207448 6535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:35.207559 6535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:35.207819 6535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:35.208119 6535 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:35.208151 6535 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:35.208175 6535 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:35.208202 6535 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:35.214499 6535 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:35281
I20240503 08:13:35.214558 6663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:35281 every 8 connection(s)
I20240503 08:13:35.214913 6535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724015, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:35.222591 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 6535
I20240503 08:13:35.222695 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:13:35.222894 6419 external_mini_cluster.cc:1428] Setting key 2df69582f16e820a5532f1e554eda16b
I20240503 08:13:35.224609 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:35.215644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40851 (local address 127.6.68.193:56607)
0503 08:13:35.215815 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:35.215844 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:35.216360 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:35.216624 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:35.216628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:35.216969 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:35.217566 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:35.217572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.218295 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:35.218296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:35.218676 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:35.218677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.218824 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:35.219541 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:35.219559 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:35.221165 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:35.223156 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:35.223165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:35.223168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:35.223475 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:35.223672 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:35.223673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:35.223674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:35.223782 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:35.224087 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:35.224093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:35.224227 (+ 134us) client_negotiation.cc:769] Sending connection context
0503 08:13:35.224324 (+ 97us) client_negotiation.cc:240] Negotiation successful
0503 08:13:35.224452 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20240503 08:13:35.224632 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:35.215678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:56607 (local address 127.6.68.254:40851)
0503 08:13:35.215743 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:35.215745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:35.215952 (+ 207us) server_negotiation.cc:407] Connection header received
0503 08:13:35.216479 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:35.216481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:35.216494 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:35.216559 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:35.217691 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.218198 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:35.218764 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.218937 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:35.221304 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:35.221309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:35.221309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:35.221317 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:35.223054 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.223553 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.223555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.223555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.223581 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.223865 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.223866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.223867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.223992 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:35.224067 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:35.224376 (+ 309us) server_negotiation.cc:299] Negotiation successful
0503 08:13:35.224490 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240503 08:13:35.225224 6664 heartbeater.cc:347] Connected to a master server at 127.6.68.254:40851
I20240503 08:13:35.225360 6664 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:35.225668 6664 heartbeater.cc:511] Master 127.6.68.254:40851 requested a full tablet report, sending...
I20240503 08:13:35.226202 6471 ts_manager.cc:194] Registered new tserver with Master: 8f67eefdb38c495e90b1063b97e3d79a (127.6.68.193:35281)
I20240503 08:13:35.227273 6471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:56607
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:13:35.252374 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40851
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:35.335750 6669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:35.335978 6669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:35.336015 6669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:35.337401 6669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:35.337457 6669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:35.337499 6669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:13:35.339130 6669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40851
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:35.339390 6669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:35.339593 6669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:35.344888 6677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:35.345497 6674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:35.346062 6675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:35.346882 6669 server_base.cc:1034] running on GCE node
I20240503 08:13:35.347110 6669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:35.347400 6669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:35.348567 6669 hybrid_clock.cc:648] HybridClock initialized: now 1714724015348543 us; error 52 us; skew 500 ppm
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724015, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:35.351336 6669 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:13:35.351855 6669 webserver.cc:460] Webserver started at http://127.6.68.194:43061/ using document root <none> and password file <none>
I20240503 08:13:35.352043 6669 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:35.352090 6669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:35.352205 6669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:35.353236 6669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "626948be63aa44dbad67508749f6fcb0"
format_stamp: "Formatted at 2024-05-03 08:13:35 on dist-test-slave-3g67"
server_key: "8ce0c734b79b4ce72d1ad445315c4fdf"
server_key_iv: "3ca19082e89cd39cc8f87dfc106ce84c"
server_key_version: "encryptionkey@0"
I20240503 08:13:35.353574 6669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "626948be63aa44dbad67508749f6fcb0"
format_stamp: "Formatted at 2024-05-03 08:13:35 on dist-test-slave-3g67"
server_key: "8ce0c734b79b4ce72d1ad445315c4fdf"
server_key_iv: "3ca19082e89cd39cc8f87dfc106ce84c"
server_key_version: "encryptionkey@0"
I20240503 08:13:35.354982 6669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:13:35.355861 6684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:35.356073 6669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:35.356161 6669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "626948be63aa44dbad67508749f6fcb0"
format_stamp: "Formatted at 2024-05-03 08:13:35 on dist-test-slave-3g67"
server_key: "8ce0c734b79b4ce72d1ad445315c4fdf"
server_key_iv: "3ca19082e89cd39cc8f87dfc106ce84c"
server_key_version: "encryptionkey@0"
I20240503 08:13:35.356232 6669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:35.374132 6669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:35.376456 6669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:35.376588 6669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:35.376816 6669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:35.377133 6669 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:35.377171 6669 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:35.377208 6669 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:35.377242 6669 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:35.383052 6669 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:36737
I20240503 08:13:35.383139 6798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:36737 every 8 connection(s)
I20240503 08:13:35.383443 6669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:13:35.387434 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 6669
I20240503 08:13:35.387559 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:13:35.387748 6419 external_mini_cluster.cc:1428] Setting key a6caed1e9db166cd0730fe6f1b7665f5
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724015, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:35.393220 6801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:35.384176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40851 (local address 127.6.68.194:51545)
0503 08:13:35.384366 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:35.384397 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:35.384865 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:35.385170 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:35.385175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:35.385536 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:35.386082 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:35.386087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.386843 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:35.386843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:35.387242 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:35.387244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.387363 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:35.388034 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:35.388059 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:35.389847 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:35.391959 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:35.391967 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:35.391971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:35.392213 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:35.392402 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:35.392403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:35.392403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:35.392459 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:35.392712 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:35.392714 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:35.392847 (+ 133us) client_negotiation.cc:769] Sending connection context
0503 08:13:35.392938 (+ 91us) client_negotiation.cc:240] Negotiation successful
0503 08:13:35.393062 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20240503 08:13:35.393237 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:35.384233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:51545 (local address 127.6.68.254:40851)
0503 08:13:35.384302 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:35.384304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:35.384526 (+ 222us) server_negotiation.cc:407] Connection header received
0503 08:13:35.384975 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:35.384977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:35.384994 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:35.385066 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:35.386207 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.386744 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:35.387344 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.387558 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:35.389973 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:35.389979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:35.389979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:35.389989 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:35.391858 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.392300 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.392301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.392302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.392336 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.392508 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.392509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.392509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.392647 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:35.392722 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:35.392986 (+ 264us) server_negotiation.cc:299] Negotiation successful
0503 08:13:35.393096 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240503 08:13:35.393756 6799 heartbeater.cc:347] Connected to a master server at 127.6.68.254:40851
I20240503 08:13:35.393896 6799 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:35.394166 6799 heartbeater.cc:511] Master 127.6.68.254:40851 requested a full tablet report, sending...
I20240503 08:13:35.394629 6471 ts_manager.cc:194] Registered new tserver with Master: 626948be63aa44dbad67508749f6fcb0 (127.6.68.194:36737)
I20240503 08:13:35.395018 6471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:51545
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:13:35.418943 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40851
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:35.502260 6804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:35.502467 6804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:35.502494 6804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:35.503960 6804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:35.504016 6804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:35.504045 6804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:13:35.505654 6804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40851
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:35.505853 6804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:35.506018 6804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:35.511058 6812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:35.511958 6810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:35.512109 6809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:35.513043 6804 server_base.cc:1034] running on GCE node
I20240503 08:13:35.513274 6804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:35.513516 6804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:35.514679 6804 hybrid_clock.cc:648] HybridClock initialized: now 1714724015514662 us; error 31 us; skew 500 ppm
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724015, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:35.522845 6804 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:13:35.523324 6804 webserver.cc:460] Webserver started at http://127.6.68.195:46641/ using document root <none> and password file <none>
I20240503 08:13:35.523531 6804 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:35.523582 6804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:35.523723 6804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:35.524785 6804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "6acd3bd773c348819300aa3a25f65a93"
format_stamp: "Formatted at 2024-05-03 08:13:35 on dist-test-slave-3g67"
server_key: "14c6a770db614c2a64d0bdc36dae2116"
server_key_iv: "c3327b1e01c2ceaf39d11a0d42d492d7"
server_key_version: "encryptionkey@0"
I20240503 08:13:35.525110 6804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "6acd3bd773c348819300aa3a25f65a93"
format_stamp: "Formatted at 2024-05-03 08:13:35 on dist-test-slave-3g67"
server_key: "14c6a770db614c2a64d0bdc36dae2116"
server_key_iv: "c3327b1e01c2ceaf39d11a0d42d492d7"
server_key_version: "encryptionkey@0"
I20240503 08:13:35.526374 6804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:13:35.527235 6819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:35.527479 6804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:35.527568 6804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "6acd3bd773c348819300aa3a25f65a93"
format_stamp: "Formatted at 2024-05-03 08:13:35 on dist-test-slave-3g67"
server_key: "14c6a770db614c2a64d0bdc36dae2116"
server_key_iv: "c3327b1e01c2ceaf39d11a0d42d492d7"
server_key_version: "encryptionkey@0"
I20240503 08:13:35.527665 6804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:35.542663 6804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:35.544926 6804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:35.545066 6804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:35.545297 6804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:35.545622 6804 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:35.545658 6804 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:35.545696 6804 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:35.545733 6804 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:35.551599 6804 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:45183
I20240503 08:13:35.551687 6932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:45183 every 8 connection(s)
I20240503 08:13:35.552004 6804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:13:35.553750 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 6804
I20240503 08:13:35.553838 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:13:35.553970 6419 external_mini_cluster.cc:1428] Setting key 3eec8d5af14b66004efa97e947840b3c
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724015, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:35.561224 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:35.552736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:41457 (local address 127.6.68.254:40851)
0503 08:13:35.552788 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:35.552790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:35.552968 (+ 178us) server_negotiation.cc:407] Connection header received
0503 08:13:35.553453 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:35.553454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:35.553464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:35.553532 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:35.554797 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.555415 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:35.556086 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.556221 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:35.558353 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:35.558357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:35.558358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:35.558365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:35.559995 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.560396 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.560398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.560398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.560425 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.560614 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.560615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.560615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.560730 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:35.560795 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:35.561031 (+ 236us) server_negotiation.cc:299] Negotiation successful
0503 08:13:35.561119 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240503 08:13:35.561345 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:35.552702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40851 (local address 127.6.68.195:41457)
0503 08:13:35.552855 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:35.552888 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:35.553360 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:35.553590 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:35.553594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:35.553950 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:35.554700 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:35.554708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.555510 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:35.555511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:35.556023 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:35.556025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.556144 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:35.556723 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:35.556741 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:35.558243 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:35.560084 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:35.560092 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:35.560095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:35.560325 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:35.560483 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:35.560484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:35.560484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:35.560538 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:35.560787 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:35.560790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:35.560901 (+ 111us) client_negotiation.cc:769] Sending connection context
0503 08:13:35.561000 (+ 99us) client_negotiation.cc:240] Negotiation successful
0503 08:13:35.561120 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20240503 08:13:35.561955 6933 heartbeater.cc:347] Connected to a master server at 127.6.68.254:40851
I20240503 08:13:35.562049 6933 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:35.562284 6933 heartbeater.cc:511] Master 127.6.68.254:40851 requested a full tablet report, sending...
I20240503 08:13:35.562680 6471 ts_manager.cc:194] Registered new tserver with Master: 6acd3bd773c348819300aa3a25f65a93 (127.6.68.195:45183)
I20240503 08:13:35.563063 6471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:41457
I20240503 08:13:35.565966 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.68.193@KRBTEST.COM
I20240503 08:13:35.588353 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:35.581023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.6.68.193:35281)
0503 08:13:35.581257 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:35.581265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:35.581277 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:13:35.581308 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:35.581310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:35.581325 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:35.581406 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:35.582069 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.582800 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:35.583407 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.583578 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:35.585699 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:35.585713 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:35.585713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:35.585720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:35.587405 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.587756 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.587759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.587759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.587788 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.587915 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.587916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.587916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.588009 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:35.588113 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:35.588165 (+ 52us) server_negotiation.cc:299] Negotiation successful
0503 08:13:35.588236 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
W20240503 08:13:35.588724 6578 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57452
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.193@KRBTEST.COM
I20240503 08:13:35.610257 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:35.603516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.6.68.193:35281)
0503 08:13:35.603577 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:35.603579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:35.603651 (+ 72us) server_negotiation.cc:407] Connection header received
0503 08:13:35.603771 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:35.603772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:35.603788 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:35.603841 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:35.604571 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.605047 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:35.605651 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:35.605760 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:35.607921 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:35.607926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:35.607926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:35.607933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:35.609417 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.609713 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.609714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.609715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.609732 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:35.609841 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:35.609842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:35.609842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:35.609924 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:35.609996 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:35.610074 (+ 78us) server_negotiation.cc:299] Negotiation successful
0503 08:13:35.610140 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240503 08:13:35.610872 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 6535
I20240503 08:13:35.615556 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 6669
I20240503 08:13:35.620659 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 6804
I20240503 08:13:35.625312 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 6439
2024-05-03T08:13:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1833 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6947](info): set up 2 sockets
May 03 08:13:35 dist-test-slave-3g67 krb5kdc[6947](info): commencing operation
krb5kdc: starting...
W20240503 08:13:36.583894 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:13:36 dist-test-slave-3g67 krb5kdc[6947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:13:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:13:36.707474 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:38123
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:38123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:36.789480 6960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:36.789707 6960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:36.789731 6960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:36.791182 6960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:36.791265 6960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:36.791280 6960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:36.791292 6960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:36.791306 6960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:36.793207 6960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:38123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:38123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:36.793486 6960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:36.793697 6960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:36.798781 6966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:36.799507 6965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:36.800154 6968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:36.800873 6960 server_base.cc:1034] running on GCE node
I20240503 08:13:36.801144 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:36.801448 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:36.802595 6960 hybrid_clock.cc:648] HybridClock initialized: now 1714724016802575 us; error 35 us; skew 500 ppm
May 03 08:13:36 dist-test-slave-3g67 krb5kdc[6947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724016, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:36.805536 6960 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:13:36.805999 6960 webserver.cc:460] Webserver started at http://127.6.68.254:37155/ using document root <none> and password file <none>
I20240503 08:13:36.806201 6960 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:36.806249 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:36.806362 6960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:36.807405 6960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "06962c766c304b66ab8111107c562715"
format_stamp: "Formatted at 2024-05-03 08:13:36 on dist-test-slave-3g67"
server_key: "acfc097d3d8ad8a39ef099334d040712"
server_key_iv: "8bcb9e91e06a345bc8fbabe5c528cc66"
server_key_version: "encryptionkey@0"
I20240503 08:13:36.807770 6960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "06962c766c304b66ab8111107c562715"
format_stamp: "Formatted at 2024-05-03 08:13:36 on dist-test-slave-3g67"
server_key: "acfc097d3d8ad8a39ef099334d040712"
server_key_iv: "8bcb9e91e06a345bc8fbabe5c528cc66"
server_key_version: "encryptionkey@0"
I20240503 08:13:36.809084 6960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:36.809939 6975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:36.810221 6960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:36.810320 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "06962c766c304b66ab8111107c562715"
format_stamp: "Formatted at 2024-05-03 08:13:36 on dist-test-slave-3g67"
server_key: "acfc097d3d8ad8a39ef099334d040712"
server_key_iv: "8bcb9e91e06a345bc8fbabe5c528cc66"
server_key_version: "encryptionkey@0"
I20240503 08:13:36.810412 6960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:36.834189 6960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:36.836406 6960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:36.836541 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:36.840447 6960 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:38123
I20240503 08:13:36.840488 7028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:38123 every 8 connection(s)
I20240503 08:13:36.840847 6960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:13:36.841809 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:36.842413 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 6960
I20240503 08:13:36.842499 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:13:36.842656 6419 external_mini_cluster.cc:1428] Setting key 86d6235717a0f289b4dab319672e2d38
I20240503 08:13:36.844866 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715: Bootstrap starting.
I20240503 08:13:36.845471 7029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:36.845729 7029 log.cc:824] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:36.846424 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715: No bootstrap required, opened a new log
May 03 08:13:36 dist-test-slave-3g67 krb5kdc[6947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:36.848564 7029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06962c766c304b66ab8111107c562715" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38123 } }
I20240503 08:13:36.848748 7029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:36.848789 7029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06962c766c304b66ab8111107c562715, State: Initialized, Role: FOLLOWER
I20240503 08:13:36.848899 7029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06962c766c304b66ab8111107c562715" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38123 } }
I20240503 08:13:36.848960 7029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:36.848984 7029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:36.849035 7029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:36.849861 7029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06962c766c304b66ab8111107c562715" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38123 } }
I20240503 08:13:36.850003 7029 leader_election.cc:302] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06962c766c304b66ab8111107c562715; no voters:
I20240503 08:13:36.850234 7029 leader_election.cc:288] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:36.850450 7035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:36.850517 7029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:36.850600 7035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [term 1 LEADER]: Becoming Leader. State: Replica: 06962c766c304b66ab8111107c562715, State: Running, Role: LEADER
I20240503 08:13:36.850699 7035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06962c766c304b66ab8111107c562715" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38123 } }
I20240503 08:13:36.851042 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06962c766c304b66ab8111107c562715. Latest consensus state: current_term: 1 leader_uuid: "06962c766c304b66ab8111107c562715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06962c766c304b66ab8111107c562715" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38123 } } }
I20240503 08:13:36.851135 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:36.851056 7034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06962c766c304b66ab8111107c562715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06962c766c304b66ab8111107c562715" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38123 } } }
I20240503 08:13:36.851202 7034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:36.851503 7039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:36.852041 7039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:36.853617 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:36.843250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.6.68.254:38123)
0503 08:13:36.843395 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:36.843409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:36.843428 (+ 19us) server_negotiation.cc:407] Connection header received
0503 08:13:36.843874 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:36.843881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:36.843931 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:36.844067 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:36.845110 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:36.845897 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:36.846506 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:36.846782 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:36.849284 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:36.849294 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:36.849294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:36.849314 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:36.852054 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:36.852620 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:36.852623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:36.852623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:36.852683 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:36.852858 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:36.852858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:36.852859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:36.853068 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:36.853244 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:36.853323 (+ 79us) server_negotiation.cc:299] Negotiation successful
0503 08:13:36.853455 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20240503 08:13:36.856770 7039 catalog_manager.cc:1347] Generated new cluster ID: ce74bdeb225945e1a2280fbb1a8cf5fb
I20240503 08:13:36.856840 7039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:13:36.868013 7039 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:36.868631 7039 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:36.874593 7039 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 06962c766c304b66ab8111107c562715: Generated new TSK 0
I20240503 08:13:36.874817 7039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:13:36.908850 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:38123
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:36.992900 7055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:36.993116 7055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:36.993158 7055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:36.994608 7055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:36.994670 7055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:36.994711 7055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:13:36.996650 7055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:38123
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:36.996948 7055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:36.997182 7055 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:37.002202 7063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:37.003011 7060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:37.003944 7061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:37.004743 7055 server_base.cc:1034] running on GCE node
I20240503 08:13:37.004935 7055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:37.005187 7055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:37.006327 7055 hybrid_clock.cc:648] HybridClock initialized: now 1714724017006310 us; error 41 us; skew 500 ppm
May 03 08:13:37 dist-test-slave-3g67 krb5kdc[6947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724017, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:37.009197 7055 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:13:37.009709 7055 webserver.cc:460] Webserver started at http://127.6.68.193:45863/ using document root <none> and password file <none>
I20240503 08:13:37.009900 7055 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:37.009949 7055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:37.010075 7055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:37.011122 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "e35ee64589f24989a0bbe6b4c0d4863c"
format_stamp: "Formatted at 2024-05-03 08:13:37 on dist-test-slave-3g67"
server_key: "5fae3bb22e4246ca33a8ccb065b1bd91"
server_key_iv: "1927775100b8080e080a25aa62c400ec"
server_key_version: "encryptionkey@0"
I20240503 08:13:37.011456 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "e35ee64589f24989a0bbe6b4c0d4863c"
format_stamp: "Formatted at 2024-05-03 08:13:37 on dist-test-slave-3g67"
server_key: "5fae3bb22e4246ca33a8ccb065b1bd91"
server_key_iv: "1927775100b8080e080a25aa62c400ec"
server_key_version: "encryptionkey@0"
I20240503 08:13:37.012804 7055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:37.013600 7070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:37.013828 7055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:37.013916 7055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "e35ee64589f24989a0bbe6b4c0d4863c"
format_stamp: "Formatted at 2024-05-03 08:13:37 on dist-test-slave-3g67"
server_key: "5fae3bb22e4246ca33a8ccb065b1bd91"
server_key_iv: "1927775100b8080e080a25aa62c400ec"
server_key_version: "encryptionkey@0"
I20240503 08:13:37.013996 7055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:37.026187 7055 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:37.028144 7055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:37.028254 7055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:37.028503 7055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:37.028817 7055 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:37.028851 7055 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:37.028894 7055 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:37.028923 7055 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:37.034487 7055 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:37655
I20240503 08:13:37.034550 7184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:37655 every 8 connection(s)
I20240503 08:13:37.034838 7055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
May 03 08:13:37 dist-test-slave-3g67 krb5kdc[6947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724017, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:37.043272 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.035641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:38573 (local address 127.6.68.254:38123)
0503 08:13:37.035702 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.035705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.035859 (+ 154us) server_negotiation.cc:407] Connection header received
0503 08:13:37.036308 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.036309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.036319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.036375 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:37.037172 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.037676 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.038199 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.038339 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.040696 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:37.040703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:37.040704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:37.040711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:37.042231 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.042568 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.042569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.042569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.042585 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.042730 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.042731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.042731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.042836 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:37.042880 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.043117 (+ 237us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.043198 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240503 08:13:37.043376 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.035571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:38123 (local address 127.6.68.193:38573)
0503 08:13:37.035747 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:37.035777 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:37.036217 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:37.036454 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:37.036458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:37.036776 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:37.037066 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.037070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.037751 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.037751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:37.038128 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.038129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.038502 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.039054 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:37.039071 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:37.040580 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:37.042311 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:37.042315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:37.042318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:37.042509 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:37.042636 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:37.042636 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:37.042637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:37.042687 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:37.042890 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:37.042892 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:37.043004 (+ 112us) client_negotiation.cc:769] Sending connection context
0503 08:13:37.043091 (+ 87us) client_negotiation.cc:240] Negotiation successful
0503 08:13:37.043208 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20240503 08:13:37.043975 7185 heartbeater.cc:347] Connected to a master server at 127.6.68.254:38123
I20240503 08:13:37.044070 7185 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:37.044277 7185 heartbeater.cc:511] Master 127.6.68.254:38123 requested a full tablet report, sending...
I20240503 08:13:37.044283 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 7055
I20240503 08:13:37.044356 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:13:37.044560 6419 external_mini_cluster.cc:1428] Setting key 7584119804686ce01982e69a4f9b97bb
I20240503 08:13:37.044852 6993 ts_manager.cc:194] Registered new tserver with Master: e35ee64589f24989a0bbe6b4c0d4863c (127.6.68.193:37655)
I20240503 08:13:37.045681 6993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:38573
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:13:37.072047 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:38123
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:37.153295 7190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:37.153509 7190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:37.153534 7190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:37.154912 7190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:37.154975 7190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:37.155002 7190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:13:37.156740 7190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:38123
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:37.156982 7190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:37.157187 7190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:37.162007 7198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:37.162652 7196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:37.163411 7195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:37.164243 7190 server_base.cc:1034] running on GCE node
I20240503 08:13:37.164446 7190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:37.164654 7190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:37.165781 7190 hybrid_clock.cc:648] HybridClock initialized: now 1714724017165773 us; error 42 us; skew 500 ppm
May 03 08:13:37 dist-test-slave-3g67 krb5kdc[6947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724017, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:37.168390 7190 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:13:37.168851 7190 webserver.cc:460] Webserver started at http://127.6.68.194:37719/ using document root <none> and password file <none>
I20240503 08:13:37.169023 7190 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:37.169059 7190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:37.169140 7190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:37.170123 7190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "c570910071d94ac8a1abc8fc3ea3600d"
format_stamp: "Formatted at 2024-05-03 08:13:37 on dist-test-slave-3g67"
server_key: "7c6679f67ebb9acd2f07830c40529ed8"
server_key_iv: "8614ac325e582badde6e04d53c20ff6d"
server_key_version: "encryptionkey@0"
I20240503 08:13:37.170401 7190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "c570910071d94ac8a1abc8fc3ea3600d"
format_stamp: "Formatted at 2024-05-03 08:13:37 on dist-test-slave-3g67"
server_key: "7c6679f67ebb9acd2f07830c40529ed8"
server_key_iv: "8614ac325e582badde6e04d53c20ff6d"
server_key_version: "encryptionkey@0"
I20240503 08:13:37.171654 7190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:13:37.172336 7205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:37.172605 7190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:37.172722 7190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "c570910071d94ac8a1abc8fc3ea3600d"
format_stamp: "Formatted at 2024-05-03 08:13:37 on dist-test-slave-3g67"
server_key: "7c6679f67ebb9acd2f07830c40529ed8"
server_key_iv: "8614ac325e582badde6e04d53c20ff6d"
server_key_version: "encryptionkey@0"
I20240503 08:13:37.172806 7190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:37.187525 7190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:37.189824 7190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:37.189960 7190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:37.190182 7190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:37.190491 7190 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:37.190524 7190 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:37.190560 7190 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:37.190593 7190 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:37.196427 7190 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:36105
I20240503 08:13:37.196475 7318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:36105 every 8 connection(s)
I20240503 08:13:37.196795 7190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
May 03 08:13:37 dist-test-slave-3g67 krb5kdc[6947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724017, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:37.205292 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.197497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:60505 (local address 127.6.68.254:38123)
0503 08:13:37.197562 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.197564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.197733 (+ 169us) server_negotiation.cc:407] Connection header received
0503 08:13:37.198188 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.198188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.198198 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.198272 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:37.199128 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.199650 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.200204 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.200400 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.202654 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:37.202658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:37.202659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:37.202666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:37.204093 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.204478 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.204479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.204479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.204505 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.204680 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.204681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.204681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.204779 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:37.204837 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.205090 (+ 253us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.205185 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240503 08:13:37.205415 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.197458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:38123 (local address 127.6.68.194:60505)
0503 08:13:37.197617 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:37.197648 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:37.198102 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:37.198360 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:37.198365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:37.198688 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:37.199016 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.199022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.199737 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.199738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:37.200122 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.200123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.200562 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.201142 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:37.201159 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:37.202547 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:37.204185 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:37.204191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:37.204195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:37.204408 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:37.204562 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:37.204562 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:37.204563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:37.204625 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:37.204835 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:37.204841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:37.204965 (+ 124us) client_negotiation.cc:769] Sending connection context
0503 08:13:37.205067 (+ 102us) client_negotiation.cc:240] Negotiation successful
0503 08:13:37.205216 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20240503 08:13:37.205857 7319 heartbeater.cc:347] Connected to a master server at 127.6.68.254:38123
I20240503 08:13:37.205965 7319 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:37.206163 7319 heartbeater.cc:511] Master 127.6.68.254:38123 requested a full tablet report, sending...
I20240503 08:13:37.206585 6993 ts_manager.cc:194] Registered new tserver with Master: c570910071d94ac8a1abc8fc3ea3600d (127.6.68.194:36105)
I20240503 08:13:37.206931 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 7190
I20240503 08:13:37.206966 6993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:60505
I20240503 08:13:37.207013 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:13:37.207194 6419 external_mini_cluster.cc:1428] Setting key 564c53dc5491b0e7052da9266a78b4f2
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:13:37.235239 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:38123
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:41445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:37.316833 7324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:37.317023 7324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:37.317049 7324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:37.318471 7324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:37.318521 7324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:37.318548 7324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:13:37.320245 7324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:41445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:38123
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:37.320448 7324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:37.320611 7324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:37.325373 7332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:37.325949 7330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:37.326658 7329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:37.327593 7324 server_base.cc:1034] running on GCE node
I20240503 08:13:37.327843 7324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:37.328075 7324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:37.329218 7324 hybrid_clock.cc:648] HybridClock initialized: now 1714724017329204 us; error 30 us; skew 500 ppm
May 03 08:13:37 dist-test-slave-3g67 krb5kdc[6947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724017, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:37.331800 7324 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:13:37.332257 7324 webserver.cc:460] Webserver started at http://127.6.68.195:42865/ using document root <none> and password file <none>
I20240503 08:13:37.332453 7324 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:37.332502 7324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:37.332629 7324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:37.333659 7324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "b1fd0e43224d4ac3afe51a1ee57b68af"
format_stamp: "Formatted at 2024-05-03 08:13:37 on dist-test-slave-3g67"
server_key: "f60055732fc7b02bf22f42444455da26"
server_key_iv: "107a63af69e01b14cf339b9d54294926"
server_key_version: "encryptionkey@0"
I20240503 08:13:37.333977 7324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "b1fd0e43224d4ac3afe51a1ee57b68af"
format_stamp: "Formatted at 2024-05-03 08:13:37 on dist-test-slave-3g67"
server_key: "f60055732fc7b02bf22f42444455da26"
server_key_iv: "107a63af69e01b14cf339b9d54294926"
server_key_version: "encryptionkey@0"
I20240503 08:13:37.335251 7324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:13:37.335989 7339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:37.336148 7324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:37.336225 7324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "b1fd0e43224d4ac3afe51a1ee57b68af"
format_stamp: "Formatted at 2024-05-03 08:13:37 on dist-test-slave-3g67"
server_key: "f60055732fc7b02bf22f42444455da26"
server_key_iv: "107a63af69e01b14cf339b9d54294926"
server_key_version: "encryptionkey@0"
I20240503 08:13:37.336288 7324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:37.353001 7324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:37.355293 7324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:37.355405 7324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:37.355655 7324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:37.355999 7324 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:37.356036 7324 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:37.356060 7324 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:37.356071 7324 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:37.361438 7324 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:34843
I20240503 08:13:37.361516 7453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:34843 every 8 connection(s)
I20240503 08:13:37.361788 7324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
May 03 08:13:37 dist-test-slave-3g67 krb5kdc[6947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724017, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:37.370101 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 7324
I20240503 08:13:37.370196 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:13:37.370395 6419 external_mini_cluster.cc:1428] Setting key dc2a7f5905ed9a01d805686e6e7ff00c
I20240503 08:13:37.371332 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.362580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:54975 (local address 127.6.68.254:38123)
0503 08:13:37.362660 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.362663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.362880 (+ 217us) server_negotiation.cc:407] Connection header received
0503 08:13:37.363337 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.363338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.363349 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.363414 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:37.364322 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.364821 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.365404 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.365552 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.368123 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:37.368129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:37.368129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:37.368136 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:37.369750 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.370206 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.370208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.370208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.370241 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.370555 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.370557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.370557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.370704 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:37.370777 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.371123 (+ 346us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.371234 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240503 08:13:37.371470 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.362511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:38123 (local address 127.6.68.195:54975)
0503 08:13:37.362748 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:37.362779 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:37.363242 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:37.363476 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:37.363480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:37.363877 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:37.364194 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.364200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.364906 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.364907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:37.365304 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.365305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.365703 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.366384 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:37.366402 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:37.368005 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:37.369855 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:37.369861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:37.369864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:37.370109 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:37.370368 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:37.370369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:37.370370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:37.370472 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:37.370774 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:37.370779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:37.370959 (+ 180us) client_negotiation.cc:769] Sending connection context
0503 08:13:37.371084 (+ 125us) client_negotiation.cc:240] Negotiation successful
0503 08:13:37.371249 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20240503 08:13:37.371980 7454 heartbeater.cc:347] Connected to a master server at 127.6.68.254:38123
I20240503 08:13:37.372069 7454 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:37.372267 7454 heartbeater.cc:511] Master 127.6.68.254:38123 requested a full tablet report, sending...
I20240503 08:13:37.372629 6993 ts_manager.cc:194] Registered new tserver with Master: b1fd0e43224d4ac3afe51a1ee57b68af (127.6.68.195:34843)
I20240503 08:13:37.372998 6993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:54975
I20240503 08:13:37.382329 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:13:37.389106 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.383778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.6.68.254:38123)
0503 08:13:37.383841 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.383843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.383976 (+ 133us) server_negotiation.cc:407] Connection header received
0503 08:13:37.384022 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.384023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.384031 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.384084 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:37.384769 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.385234 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.385755 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.385854 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.386884 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:37.386887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:37.386888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:37.386892 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:37.388160 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.388486 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.388487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.388487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.388504 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.388625 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.388626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.388626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.388740 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:37.388818 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.388972 (+ 154us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.389035 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:13:37.391505 6993 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:13:37.393065 6993 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:13:37.398367 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.395691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:34843 (local address 127.0.0.1:34344)
0503 08:13:37.395916 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:37.395925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:37.395972 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:37.396265 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:37.396266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:37.396268 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:37.396464 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.396466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.397375 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.397376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:37.398046 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.398048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.398121 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.398165 (+ 44us) client_negotiation.cc:769] Sending connection context
0503 08:13:37.398220 (+ 55us) client_negotiation.cc:240] Negotiation successful
0503 08:13:37.398278 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":67,"threads_started":1}
I20240503 08:13:37.399019 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.395477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:36105 (local address 127.0.0.1:58428)
0503 08:13:37.395871 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:37.395882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:37.395947 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:37.396217 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:37.396218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:37.396219 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:37.396449 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.396452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.397807 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.397808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:37.398788 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.398791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.398864 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.398867 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:13:37.398898 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:13:37.398945 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":2,"thread_start_us":46,"threads_started":1}
I20240503 08:13:37.399189 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.395477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:37655 (local address 127.0.0.1:59702)
0503 08:13:37.395764 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:37.395794 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:37.395884 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:37.396159 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:37.396161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:37.396164 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:37.396599 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.396600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.397925 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.397926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:37.398977 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.398980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.399042 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.399045 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:13:37.399075 (+ 30us) client_negotiation.cc:240] Negotiation successful
0503 08:13:37.399118 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":89,"threads_started":1}
I20240503 08:13:37.399472 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.395854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.6.68.195:34843)
0503 08:13:37.396095 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.396102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.396111 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:13:37.396146 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.396148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.396162 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.396262 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:37.396557 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.397279 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.398157 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.399259 (+ 1102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.399324 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.399359 (+ 35us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.399409 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20240503 08:13:37.399699 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.395578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58428 (local address 127.6.68.194:36105)
0503 08:13:37.395867 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.395874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.396007 (+ 133us) server_negotiation.cc:407] Connection header received
0503 08:13:37.396050 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.396052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.396065 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.396130 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:37.396540 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.397698 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.398914 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.399472 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.399515 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.399546 (+ 31us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.399586 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20240503 08:13:37.400058 7388 tablet_service.cc:1467] Processing CreateTablet for tablet 50c38efa04624c14bb354f90dd377bd6 (DEFAULT_TABLE table=test-table [id=4eb45da801044a0ab1e18914d66c0280]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.400316 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.395578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.6.68.193:37655)
0503 08:13:37.395770 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.395778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.395979 (+ 201us) server_negotiation.cc:407] Connection header received
0503 08:13:37.396023 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.396025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.396045 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.396134 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:37.396688 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.397842 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.399369 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.400108 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.400149 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.400181 (+ 32us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.400228 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":105,"threads_started":1}
I20240503 08:13:37.400236 7253 tablet_service.cc:1467] Processing CreateTablet for tablet 50c38efa04624c14bb354f90dd377bd6 (DEFAULT_TABLE table=test-table [id=4eb45da801044a0ab1e18914d66c0280]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.400377 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c38efa04624c14bb354f90dd377bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.400530 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c38efa04624c14bb354f90dd377bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.400806 7118 tablet_service.cc:1467] Processing CreateTablet for tablet 50c38efa04624c14bb354f90dd377bd6 (DEFAULT_TABLE table=test-table [id=4eb45da801044a0ab1e18914d66c0280]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.401115 7118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c38efa04624c14bb354f90dd377bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.403084 7471 tablet_bootstrap.cc:492] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af: Bootstrap starting.
I20240503 08:13:37.403612 7471 tablet_bootstrap.cc:654] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.403754 7472 tablet_bootstrap.cc:492] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d: Bootstrap starting.
I20240503 08:13:37.404112 7471 log.cc:824] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:37.404258 7473 tablet_bootstrap.cc:492] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c: Bootstrap starting.
I20240503 08:13:37.404620 7472 tablet_bootstrap.cc:654] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.404805 7473 tablet_bootstrap.cc:654] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.404919 7472 log.cc:824] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:37.404953 7471 tablet_bootstrap.cc:492] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af: No bootstrap required, opened a new log
I20240503 08:13:37.405012 7471 ts_tablet_manager.cc:1367] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:37.405097 7473 log.cc:824] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:37.405624 7472 tablet_bootstrap.cc:492] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d: No bootstrap required, opened a new log
I20240503 08:13:37.405691 7472 ts_tablet_manager.cc:1367] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:37.405777 7473 tablet_bootstrap.cc:492] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c: No bootstrap required, opened a new log
I20240503 08:13:37.405846 7473 ts_tablet_manager.cc:1367] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:37.407529 7471 raft_consensus.cc:348] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ee64589f24989a0bbe6b4c0d4863c" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 37655 } } peers { permanent_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34843 } } peers { permanent_uuid: "c570910071d94ac8a1abc8fc3ea3600d" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36105 } }
I20240503 08:13:37.407720 7471 raft_consensus.cc:374] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.407758 7471 raft_consensus.cc:729] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1fd0e43224d4ac3afe51a1ee57b68af, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.407824 7473 raft_consensus.cc:348] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ee64589f24989a0bbe6b4c0d4863c" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 37655 } } peers { permanent_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34843 } } peers { permanent_uuid: "c570910071d94ac8a1abc8fc3ea3600d" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36105 } }
I20240503 08:13:37.407879 7471 consensus_queue.cc:260] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ee64589f24989a0bbe6b4c0d4863c" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 37655 } } peers { permanent_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34843 } } peers { permanent_uuid: "c570910071d94ac8a1abc8fc3ea3600d" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36105 } }
I20240503 08:13:37.407958 7473 raft_consensus.cc:374] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.407986 7473 raft_consensus.cc:729] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35ee64589f24989a0bbe6b4c0d4863c, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.408102 7473 consensus_queue.cc:260] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ee64589f24989a0bbe6b4c0d4863c" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 37655 } } peers { permanent_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34843 } } peers { permanent_uuid: "c570910071d94ac8a1abc8fc3ea3600d" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36105 } }
I20240503 08:13:37.408154 7471 ts_tablet_manager.cc:1398] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:37.408282 7454 heartbeater.cc:503] Master 127.6.68.254:38123 was elected leader, sending a full tablet report...
I20240503 08:13:37.408360 7473 ts_tablet_manager.cc:1398] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:37.408375 7472 raft_consensus.cc:348] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ee64589f24989a0bbe6b4c0d4863c" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 37655 } } peers { permanent_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34843 } } peers { permanent_uuid: "c570910071d94ac8a1abc8fc3ea3600d" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36105 } }
I20240503 08:13:37.408481 7472 raft_consensus.cc:374] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.408502 7472 raft_consensus.cc:729] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c570910071d94ac8a1abc8fc3ea3600d, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.408499 7185 heartbeater.cc:503] Master 127.6.68.254:38123 was elected leader, sending a full tablet report...
I20240503 08:13:37.408612 7472 consensus_queue.cc:260] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ee64589f24989a0bbe6b4c0d4863c" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 37655 } } peers { permanent_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34843 } } peers { permanent_uuid: "c570910071d94ac8a1abc8fc3ea3600d" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36105 } }
I20240503 08:13:37.408849 7472 ts_tablet_manager.cc:1398] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:37.408991 7319 heartbeater.cc:503] Master 127.6.68.254:38123 was elected leader, sending a full tablet report...
W20240503 08:13:37.447399 7320 tablet.cc:2369] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:13:37.535702 7186 tablet.cc:2369] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:13:37.612329 7455 tablet.cc:2369] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:13:37.857424 7477 raft_consensus.cc:482] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:37.857574 7477 raft_consensus.cc:504] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ee64589f24989a0bbe6b4c0d4863c" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 37655 } } peers { permanent_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34843 } } peers { permanent_uuid: "c570910071d94ac8a1abc8fc3ea3600d" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36105 } }
I20240503 08:13:37.857925 7477 leader_election.cc:288] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e35ee64589f24989a0bbe6b4c0d4863c (127.6.68.193:37655), c570910071d94ac8a1abc8fc3ea3600d (127.6.68.194:36105)
I20240503 08:13:37.861016 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.858150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:37655 (local address 127.6.68.195:48945)
0503 08:13:37.858219 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:37.858231 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:37.858279 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:37.858466 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:37.858467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:37.858469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:37.858727 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.858730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.859729 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.859730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:37.860712 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.860715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.860804 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.860808 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:13:37.860844 (+ 36us) client_negotiation.cc:240] Negotiation successful
0503 08:13:37.860889 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20240503 08:13:37.861025 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.858163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:36105 (local address 127.6.68.195:44673)
0503 08:13:37.858293 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:37.858301 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:37.858362 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:37.858519 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:37.858520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:37.858535 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:37.858756 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.858757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.859771 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.859772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:37.860712 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:37.860715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.860804 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.860808 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:13:37.860844 (+ 36us) client_negotiation.cc:240] Negotiation successful
0503 08:13:37.860881 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":12,"thread_start_us":67,"threads_started":1}
I20240503 08:13:37.861459 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.858253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:44673 (local address 127.6.68.194:36105)
0503 08:13:37.858299 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.858301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.858356 (+ 55us) server_negotiation.cc:407] Connection header received
0503 08:13:37.858435 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.858436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.858445 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.858504 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:37.858825 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.859612 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.860818 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.861318 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.861326 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.861350 (+ 24us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.861383 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:13:37.861449 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.858261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:48945 (local address 127.6.68.193:37655)
0503 08:13:37.858311 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.858313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.858321 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:13:37.858353 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.858354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.858365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.858447 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:37.858824 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.859610 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.860818 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.861318 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.861326 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.861350 (+ 24us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.861381 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240503 08:13:37.861765 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c38efa04624c14bb354f90dd377bd6" candidate_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ee64589f24989a0bbe6b4c0d4863c" is_pre_election: true
I20240503 08:13:37.861768 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c38efa04624c14bb354f90dd377bd6" candidate_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c570910071d94ac8a1abc8fc3ea3600d" is_pre_election: true
I20240503 08:13:37.861912 7138 raft_consensus.cc:2454] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1fd0e43224d4ac3afe51a1ee57b68af in term 0.
I20240503 08:13:37.861928 7273 raft_consensus.cc:2454] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1fd0e43224d4ac3afe51a1ee57b68af in term 0.
I20240503 08:13:37.862150 7341 leader_election.cc:302] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1fd0e43224d4ac3afe51a1ee57b68af, c570910071d94ac8a1abc8fc3ea3600d; no voters:
I20240503 08:13:37.862334 7477 raft_consensus.cc:2781] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:37.862409 7477 raft_consensus.cc:482] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:37.862427 7477 raft_consensus.cc:3037] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.863137 7477 raft_consensus.cc:504] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ee64589f24989a0bbe6b4c0d4863c" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 37655 } } peers { permanent_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34843 } } peers { permanent_uuid: "c570910071d94ac8a1abc8fc3ea3600d" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36105 } }
I20240503 08:13:37.863270 7477 leader_election.cc:288] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [CANDIDATE]: Term 1 election: Requested vote from peers e35ee64589f24989a0bbe6b4c0d4863c (127.6.68.193:37655), c570910071d94ac8a1abc8fc3ea3600d (127.6.68.194:36105)
I20240503 08:13:37.863461 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c38efa04624c14bb354f90dd377bd6" candidate_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c570910071d94ac8a1abc8fc3ea3600d"
I20240503 08:13:37.863461 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c38efa04624c14bb354f90dd377bd6" candidate_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ee64589f24989a0bbe6b4c0d4863c"
I20240503 08:13:37.863533 7273 raft_consensus.cc:3037] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.863538 7138 raft_consensus.cc:3037] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.864243 7138 raft_consensus.cc:2454] T 50c38efa04624c14bb354f90dd377bd6 P e35ee64589f24989a0bbe6b4c0d4863c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1fd0e43224d4ac3afe51a1ee57b68af in term 1.
I20240503 08:13:37.864248 7273 raft_consensus.cc:2454] T 50c38efa04624c14bb354f90dd377bd6 P c570910071d94ac8a1abc8fc3ea3600d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1fd0e43224d4ac3afe51a1ee57b68af in term 1.
I20240503 08:13:37.864444 7342 leader_election.cc:302] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1fd0e43224d4ac3afe51a1ee57b68af, e35ee64589f24989a0bbe6b4c0d4863c; no voters:
I20240503 08:13:37.864562 7477 raft_consensus.cc:2781] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:37.864745 7477 raft_consensus.cc:686] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [term 1 LEADER]: Becoming Leader. State: Replica: b1fd0e43224d4ac3afe51a1ee57b68af, State: Running, Role: LEADER
I20240503 08:13:37.864853 7477 consensus_queue.cc:237] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ee64589f24989a0bbe6b4c0d4863c" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 37655 } } peers { permanent_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34843 } } peers { permanent_uuid: "c570910071d94ac8a1abc8fc3ea3600d" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36105 } }
I20240503 08:13:37.865732 6993 catalog_manager.cc:5484] T 50c38efa04624c14bb354f90dd377bd6 P b1fd0e43224d4ac3afe51a1ee57b68af reported cstate change: term changed from 0 to 1, leader changed from <none> to b1fd0e43224d4ac3afe51a1ee57b68af (127.6.68.195). New cstate: current_term: 1 leader_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35ee64589f24989a0bbe6b4c0d4863c" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 37655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1fd0e43224d4ac3afe51a1ee57b68af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c570910071d94ac8a1abc8fc3ea3600d" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36105 } health_report { overall_health: UNKNOWN } } }
May 03 08:13:37 dist-test-slave-3g67 krb5kdc[6947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.193@KRBTEST.COM
I20240503 08:13:37.954295 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.947361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.6.68.193:37655)
0503 08:13:37.947454 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.947457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.947531 (+ 74us) server_negotiation.cc:407] Connection header received
0503 08:13:37.947608 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.947609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.947637 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.947682 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:37.948314 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.948771 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.949311 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.949502 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.951668 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:37.951683 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:37.951683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:37.951690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:37.953297 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.953615 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.953617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.953617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.953648 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.953788 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.953788 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.953789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.953888 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:37.954018 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.954078 (+ 60us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.954186 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
May 03 08:13:37 dist-test-slave-3g67 krb5kdc[6947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 03 08:13:37 dist-test-slave-3g67 krb5kdc[6947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.68.193@KRBTEST.COM
I20240503 08:13:37.976210 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.969335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.6.68.193:37655)
0503 08:13:37.969423 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.969425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.969502 (+ 77us) server_negotiation.cc:407] Connection header received
0503 08:13:37.969586 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.969588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.969599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.969647 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:37.970391 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.970840 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.971431 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.971651 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.973705 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:37.973711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:37.973711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:37.973717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:37.975119 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.975478 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.975480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.975480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.975504 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.975660 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.975661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.975661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.975750 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:37.975842 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.976006 (+ 164us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.976102 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
May 03 08:13:37 dist-test-slave-3g67 krb5kdc[6947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 03 08:13:37 dist-test-slave-3g67 krb5kdc[6947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.193@KRBTEST.COM
I20240503 08:13:37.998636 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:37.991806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.6.68.193:37655)
0503 08:13:37.991890 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:37.991893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:37.991977 (+ 84us) server_negotiation.cc:407] Connection header received
0503 08:13:37.992076 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:37.992077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:37.992087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:37.992134 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:37.992885 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.993360 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:37.993918 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:37.994141 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:37.996240 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:37.996245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:37.996246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:37.996252 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:37.997702 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.998014 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.998015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.998015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.998038 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:37.998165 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:37.998166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:37.998167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:37.998248 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:37.998333 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:37.998441 (+ 108us) server_negotiation.cc:299] Negotiation successful
0503 08:13:37.998525 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20240503 08:13:37.999099 7098 tablet_service.cc:2908] Rejecting scan request for tablet 50c38efa04624c14bb354f90dd377bd6: Uninitialized: clean time has not yet been initialized
I20240503 08:13:37.999930 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 7055
I20240503 08:13:38.005574 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 7190
I20240503 08:13:38.010355 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 7324
I20240503 08:13:38.015702 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 6960
2024-05-03T08:13:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2389 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:13:38 dist-test-slave-3g67 krb5kdc[7495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:13:38 dist-test-slave-3g67 krb5kdc[7495](info): set up 2 sockets
May 03 08:13:38 dist-test-slave-3g67 krb5kdc[7495](info): commencing operation
krb5kdc: starting...
W20240503 08:13:38.972139 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:13:39 dist-test-slave-3g67 krb5kdc[7495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:13:39.094584 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40675
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:42715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:39.174734 7508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:39.174935 7508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:39.174961 7508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:39.176393 7508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:39.176450 7508 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240503 08:13:39.176462 7508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:39.176476 7508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:39.176487 7508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:39.176498 7508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:39.178164 7508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:42715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40675
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:39.178402 7508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:39.178592 7508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:39.183295 7516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.184038 7513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.184890 7514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:39.185675 7508 server_base.cc:1034] running on GCE node
I20240503 08:13:39.185920 7508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:39.186240 7508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:39.187395 7508 hybrid_clock.cc:648] HybridClock initialized: now 1714724019187373 us; error 38 us; skew 500 ppm
May 03 08:13:39 dist-test-slave-3g67 krb5kdc[7495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:39.189903 7508 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:13:39.190337 7508 webserver.cc:460] Webserver started at http://127.6.68.254:39457/ using document root <none> and password file <none>
I20240503 08:13:39.190524 7508 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:39.190575 7508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:39.190688 7508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:39.191809 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "801c42b2699945be950cf3da7cf0b139"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "18a14b92543b90225c4093f13aef5733"
server_key_iv: "b7b7d8902458470c3347229ba275bada"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.192130 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "801c42b2699945be950cf3da7cf0b139"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "18a14b92543b90225c4093f13aef5733"
server_key_iv: "b7b7d8902458470c3347229ba275bada"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.193434 7508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:39.194212 7523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.194433 7508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:39.194530 7508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "801c42b2699945be950cf3da7cf0b139"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "18a14b92543b90225c4093f13aef5733"
server_key_iv: "b7b7d8902458470c3347229ba275bada"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.194591 7508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:39.234807 7508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:39.236786 7508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:39.236922 7508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:39.241293 7508 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:40675
I20240503 08:13:39.241539 7586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:40675 every 8 connection(s)
I20240503 08:13:39.241694 7508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:13:39.242290 7587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:39.244482 7587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139: Bootstrap starting.
I20240503 08:13:39.245054 7587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:39.245298 7587 log.cc:824] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:39.246030 7587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139: No bootstrap required, opened a new log
I20240503 08:13:39.247444 7587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801c42b2699945be950cf3da7cf0b139" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40675 } }
I20240503 08:13:39.247574 7587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:39.247592 7587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801c42b2699945be950cf3da7cf0b139, State: Initialized, Role: FOLLOWER
I20240503 08:13:39.247771 7587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801c42b2699945be950cf3da7cf0b139" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40675 } }
I20240503 08:13:39.247855 7587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:39.247895 7587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:39.247951 7587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:39.248796 7587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801c42b2699945be950cf3da7cf0b139" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40675 } }
I20240503 08:13:39.248901 7587 leader_election.cc:302] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 801c42b2699945be950cf3da7cf0b139; no voters:
I20240503 08:13:39.249064 7587 leader_election.cc:288] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:39.249156 7590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:39.249329 7590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [term 1 LEADER]: Becoming Leader. State: Replica: 801c42b2699945be950cf3da7cf0b139, State: Running, Role: LEADER
I20240503 08:13:39.249323 7587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:39.249650 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 7508
I20240503 08:13:39.249732 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:13:39.249888 6419 external_mini_cluster.cc:1428] Setting key 328b61b87e11ba08766ab9db10c57d19
I20240503 08:13:39.250100 7590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801c42b2699945be950cf3da7cf0b139" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40675 } }
I20240503 08:13:39.250702 7591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "801c42b2699945be950cf3da7cf0b139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801c42b2699945be950cf3da7cf0b139" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40675 } } }
I20240503 08:13:39.250794 7591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:39.251077 7592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 801c42b2699945be950cf3da7cf0b139. Latest consensus state: current_term: 1 leader_uuid: "801c42b2699945be950cf3da7cf0b139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801c42b2699945be950cf3da7cf0b139" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40675 } } }
I20240503 08:13:39.251214 7592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139 [sys.catalog]: This master's current role is: LEADER
W20240503 08:13:39.251745 7608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:13:39.251807 7608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:13:39.251854 7603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:39.252041 7603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:39.253509 7603 catalog_manager.cc:1347] Generated new cluster ID: 4c2f1021acbe472584cf0c85e9dcbe22
I20240503 08:13:39.253566 7603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 03 08:13:39 dist-test-slave-3g67 krb5kdc[7495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:39.261240 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.250587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.6.68.254:40675)
0503 08:13:39.250754 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:39.250768 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:39.251031 (+ 263us) server_negotiation.cc:407] Connection header received
0503 08:13:39.251191 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:39.251198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:39.251257 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:39.251434 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:39.252585 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.253697 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.254480 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.254720 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.257341 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:39.257351 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:39.257352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:39.257380 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:39.259686 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.260102 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.260105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.260106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.260177 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.260343 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.260343 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.260344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.260596 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:39.260721 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:39.260870 (+ 149us) server_negotiation.cc:299] Negotiation successful
0503 08:13:39.261015 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20240503 08:13:39.262630 7603 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:39.263126 7603 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:39.267658 7603 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139: Generated new TSK 0
I20240503 08:13:39.267850 7603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:13:39.299742 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40675
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:42715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:39.383401 7613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:39.383613 7613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:39.383678 7613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:39.385035 7613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:39.385090 7613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:39.385123 7613 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240503 08:13:39.385149 7613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:13:39.386837 7613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:42715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.68.254:40675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:39.387092 7613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:39.387292 7613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:39.392818 7618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.393461 7619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.394340 7621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:39.395076 7613 server_base.cc:1034] running on GCE node
I20240503 08:13:39.395296 7613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:39.395563 7613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:39.396733 7613 hybrid_clock.cc:648] HybridClock initialized: now 1714724019396722 us; error 31 us; skew 500 ppm
May 03 08:13:39 dist-test-slave-3g67 krb5kdc[7495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:39.400036 7613 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:13:39.400579 7613 webserver.cc:460] Webserver started at http://127.6.68.193:46307/ using document root <none> and password file <none>
I20240503 08:13:39.400789 7613 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:39.400839 7613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:39.400987 7613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:39.402333 7613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "4d96cc49c65b477c960e71873f824c57"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "e403bfcf30e1a8ce8614291acc6d544a"
server_key_iv: "39e27d1a7563e691b97e74c158b918a5"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.402699 7613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "4d96cc49c65b477c960e71873f824c57"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "e403bfcf30e1a8ce8614291acc6d544a"
server_key_iv: "39e27d1a7563e691b97e74c158b918a5"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.404462 7613 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240503 08:13:39.405289 7628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.405464 7613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:39.405550 7613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "4d96cc49c65b477c960e71873f824c57"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "e403bfcf30e1a8ce8614291acc6d544a"
server_key_iv: "39e27d1a7563e691b97e74c158b918a5"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.405620 7613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:39.416790 7613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:39.418978 7613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:39.419111 7613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:39.419680 7613 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:39.419724 7613 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.419749 7613 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:39.419760 7613 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.426146 7613 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:39487
I20240503 08:13:39.426224 7743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:39487 every 8 connection(s)
I20240503 08:13:39.426523 7613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
May 03 08:13:39 dist-test-slave-3g67 krb5kdc[7495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:39.429811 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.420127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:56225 (local address 127.6.68.254:40675)
0503 08:13:39.420202 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:39.420205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:39.420354 (+ 149us) server_negotiation.cc:407] Connection header received
0503 08:13:39.420903 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:39.420905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:39.420935 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:39.421017 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:39.422176 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.423051 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.423915 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.424147 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.427244 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:39.427249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:39.427250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:39.427257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:39.428635 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.429014 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.429014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.429014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.429030 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.429207 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.429208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.429208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.429338 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:39.429406 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:39.429642 (+ 236us) server_negotiation.cc:299] Negotiation successful
0503 08:13:39.429739 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240503 08:13:39.429889 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.420082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40675 (local address 127.6.68.193:56225)
0503 08:13:39.420242 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:39.420271 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:39.420778 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:39.421084 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:39.421089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:39.421549 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:39.422009 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.422016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.423165 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.423166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:39.423824 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.423827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.423993 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.424742 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:39.424765 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:39.427135 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:39.428719 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.428725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.428729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.428941 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.429073 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.429073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.429074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.429124 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.429411 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:39.429414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:39.429527 (+ 113us) client_negotiation.cc:769] Sending connection context
0503 08:13:39.429612 (+ 85us) client_negotiation.cc:240] Negotiation successful
0503 08:13:39.429728 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":98,"threads_started":1}
I20240503 08:13:39.430451 7744 heartbeater.cc:347] Connected to a master server at 127.6.68.254:40675
I20240503 08:13:39.430536 7744 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:39.430748 7744 heartbeater.cc:511] Master 127.6.68.254:40675 requested a full tablet report, sending...
I20240503 08:13:39.431308 7535 ts_manager.cc:194] Registered new tserver with Master: 4d96cc49c65b477c960e71873f824c57 (127.6.68.193:39487)
I20240503 08:13:39.432350 7535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:56225
I20240503 08:13:39.434968 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 7613
I20240503 08:13:39.435050 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:13:39.435213 6419 external_mini_cluster.cc:1428] Setting key ce2995e51acb82e4ac3e0330e6477e60
I20240503 08:13:39.437057 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.431503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40675 (local address 127.6.68.193:44555)
0503 08:13:39.431694 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:39.431705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:39.431775 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:39.431896 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:39.431897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:39.432061 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:39.432316 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.432318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.433182 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.433183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:39.433593 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.433594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.433654 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.434118 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:39.434121 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:39.434384 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:39.436185 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.436186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.436186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.436328 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.436479 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.436479 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.436480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.436501 (+ 21us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.436733 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:39.436735 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:39.436789 (+ 54us) client_negotiation.cc:769] Sending connection context
0503 08:13:39.436860 (+ 71us) client_negotiation.cc:240] Negotiation successful
0503 08:13:39.436958 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20240503 08:13:39.437127 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.431561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:44555 (local address 127.6.68.254:40675)
0503 08:13:39.431603 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:39.431605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:39.431770 (+ 165us) server_negotiation.cc:407] Connection header received
0503 08:13:39.431832 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:39.431833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:39.431842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:39.431883 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:39.432401 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.433085 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.433665 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.433782 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.434504 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:39.434508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:39.434508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:39.434514 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:39.436104 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.436395 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.436397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.436397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.436421 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.436534 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.436535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.436535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.436646 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:39.436709 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:39.436895 (+ 186us) server_negotiation.cc:299] Negotiation successful
0503 08:13:39.436989 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:13:39.463606 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40675
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:42715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:39.545141 7753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:39.545363 7753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:39.545403 7753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:39.546782 7753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:39.546839 7753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:39.546873 7753 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240503 08:13:39.546900 7753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:13:39.548686 7753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:42715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.68.254:40675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:39.548954 7753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:39.549172 7753 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:39.554229 7758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.554680 7759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.555423 7761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:39.556382 7753 server_base.cc:1034] running on GCE node
I20240503 08:13:39.556613 7753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:39.556926 7753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:39.558109 7753 hybrid_clock.cc:648] HybridClock initialized: now 1714724019558087 us; error 36 us; skew 500 ppm
May 03 08:13:39 dist-test-slave-3g67 krb5kdc[7495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:39.560756 7753 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:13:39.561209 7753 webserver.cc:460] Webserver started at http://127.6.68.194:34373/ using document root <none> and password file <none>
I20240503 08:13:39.561416 7753 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:39.561466 7753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:39.561596 7753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:39.562647 7753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "06207493c36d45ed906a6d2f90721a6b"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "96df644249bd50b67221d5f6121e92c0"
server_key_iv: "312e95dd80dce168063e0e242c5c4a20"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.562973 7753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "06207493c36d45ed906a6d2f90721a6b"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "96df644249bd50b67221d5f6121e92c0"
server_key_iv: "312e95dd80dce168063e0e242c5c4a20"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.564357 7753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:13:39.565136 7768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.565412 7753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:39.565500 7753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "06207493c36d45ed906a6d2f90721a6b"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "96df644249bd50b67221d5f6121e92c0"
server_key_iv: "312e95dd80dce168063e0e242c5c4a20"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.565572 7753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:39.585965 7753 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:39.588261 7753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:39.588402 7753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:39.588972 7753 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:39.589025 7753 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.589063 7753 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:39.589093 7753 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.595166 7753 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:34989
I20240503 08:13:39.595240 7884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:34989 every 8 connection(s)
I20240503 08:13:39.595521 7753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
May 03 08:13:39 dist-test-slave-3g67 krb5kdc[7495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:39.598613 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 7753
I20240503 08:13:39.598701 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:13:39.598903 6419 external_mini_cluster.cc:1428] Setting key bcf54e6863977a9c580bffdc3834b8ea
I20240503 08:13:39.599088 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.589406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:36987 (local address 127.6.68.254:40675)
0503 08:13:39.589483 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:39.589486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:39.589719 (+ 233us) server_negotiation.cc:407] Connection header received
0503 08:13:39.590315 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:39.590317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:39.590330 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:39.590421 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:39.591504 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.592389 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.593322 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.593505 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.596392 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:39.596398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:39.596398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:39.596406 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:39.597739 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.598119 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.598120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.598121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.598148 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.598335 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.598336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.598337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.598451 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:39.598511 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:39.598877 (+ 366us) server_negotiation.cc:299] Negotiation successful
0503 08:13:39.598991 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240503 08:13:39.599090 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.589432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40675 (local address 127.6.68.194:36987)
0503 08:13:39.589598 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:39.589628 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:39.590187 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:39.590503 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:39.590508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:39.590950 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:39.591391 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.591397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.592485 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.592486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:39.593221 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.593225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.593400 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.594470 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:39.594485 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:39.596288 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:39.597826 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.597833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.597836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.598039 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.598222 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.598222 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.598223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.598279 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.598506 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:39.598511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:39.598633 (+ 122us) client_negotiation.cc:769] Sending connection context
0503 08:13:39.598750 (+ 117us) client_negotiation.cc:240] Negotiation successful
0503 08:13:39.598890 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20240503 08:13:39.599747 7885 heartbeater.cc:347] Connected to a master server at 127.6.68.254:40675
I20240503 08:13:39.599849 7885 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:39.600076 7885 heartbeater.cc:511] Master 127.6.68.254:40675 requested a full tablet report, sending...
I20240503 08:13:39.600471 7539 ts_manager.cc:194] Registered new tserver with Master: 06207493c36d45ed906a6d2f90721a6b (127.6.68.194:34989)
I20240503 08:13:39.601012 7539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:36987
I20240503 08:13:39.606554 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.601055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:33831 (local address 127.6.68.254:40675)
0503 08:13:39.601098 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:39.601100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:39.601140 (+ 40us) server_negotiation.cc:407] Connection header received
0503 08:13:39.601228 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:39.601229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:39.601241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:39.601315 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:39.601862 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.602319 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.603052 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.603206 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.604123 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:39.604126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:39.604126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:39.604132 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:39.605570 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.605932 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.605933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.605933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.605948 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.606088 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.606088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.606089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.606198 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:39.606243 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:39.606421 (+ 178us) server_negotiation.cc:299] Negotiation successful
0503 08:13:39.606485 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:13:39.606644 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.600893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40675 (local address 127.6.68.194:33831)
0503 08:13:39.601067 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:39.601079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:39.601166 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:39.601319 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:39.601320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:39.601500 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:39.601736 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.601738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.602396 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.602397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:39.602958 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.602961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.603036 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.603606 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:39.603612 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:39.604020 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:39.605656 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.605657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.605657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.605859 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.606001 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.606001 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.606002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.606035 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.606253 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:39.606254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:39.606310 (+ 56us) client_negotiation.cc:769] Sending connection context
0503 08:13:39.606400 (+ 90us) client_negotiation.cc:240] Negotiation successful
0503 08:13:39.606515 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:13:39.627440 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40675
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:42715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:39.707268 7894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:39.707479 7894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:39.707504 7894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:39.709013 7894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:39.709069 7894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:39.709089 7894 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240503 08:13:39.709106 7894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:13:39.710819 7894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:42715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.68.254:40675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:39.711053 7894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:39.711256 7894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:39.716462 7899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.717057 7900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.717762 7902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:39.718416 7894 server_base.cc:1034] running on GCE node
I20240503 08:13:39.718636 7894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:39.718854 7894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:39.719991 7894 hybrid_clock.cc:648] HybridClock initialized: now 1714724019719981 us; error 27 us; skew 500 ppm
May 03 08:13:39 dist-test-slave-3g67 krb5kdc[7495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:39.722616 7894 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:13:39.723073 7894 webserver.cc:460] Webserver started at http://127.6.68.195:35871/ using document root <none> and password file <none>
I20240503 08:13:39.723253 7894 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:39.723299 7894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:39.723413 7894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:39.724519 7894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "3cf174e6bc884f0cba6f65cb0c336e5e"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "2877ed7fa70191002338976d1381bfc5"
server_key_iv: "7a9ba2c70922c1a27008c8cbfccfbc38"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.724843 7894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "3cf174e6bc884f0cba6f65cb0c336e5e"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "2877ed7fa70191002338976d1381bfc5"
server_key_iv: "7a9ba2c70922c1a27008c8cbfccfbc38"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.726056 7894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:13:39.726699 7909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.726883 7894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:13:39.726960 7894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "3cf174e6bc884f0cba6f65cb0c336e5e"
format_stamp: "Formatted at 2024-05-03 08:13:39 on dist-test-slave-3g67"
server_key: "2877ed7fa70191002338976d1381bfc5"
server_key_iv: "7a9ba2c70922c1a27008c8cbfccfbc38"
server_key_version: "encryptionkey@0"
I20240503 08:13:39.727025 7894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:39.788234 7894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:39.790355 7894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:39.790479 7894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:39.791007 7894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:39.791061 7894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.791096 7894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:39.791116 7894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.797158 7894 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:35547
I20240503 08:13:39.797225 8024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:35547 every 8 connection(s)
May 03 08:13:39 dist-test-slave-3g67 krb5kdc[7495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:39.797515 7894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:13:39.800510 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.791407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:33085 (local address 127.6.68.254:40675)
0503 08:13:39.791484 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:39.791488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:39.791632 (+ 144us) server_negotiation.cc:407] Connection header received
0503 08:13:39.792165 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:39.792167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:39.792182 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:39.792260 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:39.793365 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.794220 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.795045 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.795259 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.798134 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:39.798140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:39.798140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:39.798147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:39.799428 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.799854 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.799855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.799855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.799876 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.800017 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.800018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.800018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.800104 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:39.800148 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:39.800379 (+ 231us) server_negotiation.cc:299] Negotiation successful
0503 08:13:39.800444 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240503 08:13:39.800628 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.791355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40675 (local address 127.6.68.195:33085)
0503 08:13:39.791512 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:39.791541 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:39.792060 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:39.792371 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:39.792378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:39.792797 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:39.793245 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.793250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.794321 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.794322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:39.794947 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.794950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.795113 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.795823 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:39.795839 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:39.798036 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:39.799508 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.799514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.799517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.799778 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.799928 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.799928 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.799928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.799981 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.800148 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:39.800151 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:39.800259 (+ 108us) client_negotiation.cc:769] Sending connection context
0503 08:13:39.800349 (+ 90us) client_negotiation.cc:240] Negotiation successful
0503 08:13:39.800463 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20240503 08:13:39.801206 8025 heartbeater.cc:347] Connected to a master server at 127.6.68.254:40675
I20240503 08:13:39.801281 8025 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:39.801474 8025 heartbeater.cc:511] Master 127.6.68.254:40675 requested a full tablet report, sending...
I20240503 08:13:39.801856 7535 ts_manager.cc:194] Registered new tserver with Master: 3cf174e6bc884f0cba6f65cb0c336e5e (127.6.68.195:35547)
I20240503 08:13:39.802254 7535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:33085
I20240503 08:13:39.802894 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 7894
I20240503 08:13:39.802963 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:13:39.803138 6419 external_mini_cluster.cc:1428] Setting key 025dc7558d2bbb2a0912bd4739ab95ef
I20240503 08:13:39.803943 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:13:39.807744 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.802284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:55095 (local address 127.6.68.254:40675)
0503 08:13:39.802384 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:39.802386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:39.802424 (+ 38us) server_negotiation.cc:407] Connection header received
0503 08:13:39.802501 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:39.802502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:39.802512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:39.802586 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:39.803102 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.803778 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.804440 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.804554 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.805462 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:39.805466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:39.805466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:39.805472 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:39.806784 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.807090 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.807092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.807092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.807116 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.807254 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.807254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.807255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.807343 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:39.807425 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:39.807541 (+ 116us) server_negotiation.cc:299] Negotiation successful
0503 08:13:39.807646 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20240503 08:13:39.807762 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.802208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40675 (local address 127.6.68.195:55095)
0503 08:13:39.802347 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:39.802359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:39.802449 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:39.802576 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:39.802577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:39.802756 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:39.803017 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.803019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.803840 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.803841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:39.804360 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.804363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.804443 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.804996 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:39.805000 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:39.805370 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:39.806873 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.806875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.806875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.807025 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.807171 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:39.807171 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:39.807172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:39.807200 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:39.807397 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:39.807398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:39.807441 (+ 43us) client_negotiation.cc:769] Sending connection context
0503 08:13:39.807514 (+ 73us) client_negotiation.cc:240] Negotiation successful
0503 08:13:39.807641 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
May 03 08:13:39 dist-test-slave-3g67 krb5kdc[7495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 03 08:13:39 dist-test-slave-3g67 krb5kdc[7495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:39.825992 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.819367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.6.68.254:40675)
0503 08:13:39.819437 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:39.819439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:39.819452 (+ 13us) server_negotiation.cc:407] Connection header received
0503 08:13:39.819525 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:39.819526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:39.819539 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:39.819588 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:39.820334 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.820810 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.821405 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.821588 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.823727 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:39.823732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:39.823732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:39.823739 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:39.825000 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.825293 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.825294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.825294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.825318 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:39.825477 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:39.825478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:39.825478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:39.825578 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:39.825674 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:39.825774 (+ 100us) server_negotiation.cc:299] Negotiation successful
0503 08:13:39.825861 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240503 08:13:39.827709 7535 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:13:39.829193 7535 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:13:39.834740 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.831444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:34989 (local address 127.0.0.1:39904)
0503 08:13:39.831558 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:39.831580 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:39.831705 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:39.831994 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:39.831996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:39.831999 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:39.832286 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.832288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.833634 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.833635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:39.834455 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.834457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.834510 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.834511 (+ 1us) client_negotiation.cc:769] Sending connection context
0503 08:13:39.834566 (+ 55us) client_negotiation.cc:240] Negotiation successful
0503 08:13:39.834607 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
I20240503 08:13:39.834757 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.831761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39487 (local address 127.0.0.1:42550)
0503 08:13:39.831885 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:39.831895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:39.831948 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:39.832214 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:39.832215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:39.832217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:39.832436 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.832438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.833503 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.833504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:39.834376 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.834378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.834458 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.834510 (+ 52us) client_negotiation.cc:769] Sending connection context
0503 08:13:39.834568 (+ 58us) client_negotiation.cc:240] Negotiation successful
0503 08:13:39.834607 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20240503 08:13:39.835168 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.831923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:35547 (local address 127.0.0.1:42178)
0503 08:13:39.832112 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:39.832121 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:39.832170 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:39.832663 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:39.832665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:39.832668 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:39.832880 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.832881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.834035 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.834036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:39.834920 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:39.834924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.835002 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.835007 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:13:39.835040 (+ 33us) client_negotiation.cc:240] Negotiation successful
0503 08:13:39.835081 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
I20240503 08:13:39.835737 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.832120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.6.68.195:35547)
0503 08:13:39.832309 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:39.832318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:39.832326 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:13:39.832361 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:39.832362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:39.832485 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:39.832567 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:39.832967 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.833945 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.835032 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.835540 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.835582 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:39.835610 (+ 28us) server_negotiation.cc:299] Negotiation successful
0503 08:13:39.835681 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20240503 08:13:39.835816 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.831559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.6.68.194:34989)
0503 08:13:39.831769 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:39.831778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:39.831788 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:13:39.831844 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:39.831846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:39.831865 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:39.832373 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:39.832456 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.833549 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.835070 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.835581 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.835660 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:39.835693 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:13:39.835734 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":122,"threads_started":1}
I20240503 08:13:39.836091 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:39.831774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.6.68.193:39487)
0503 08:13:39.831972 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:39.831980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:39.831991 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:13:39.832028 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:39.832029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:39.832042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:39.832115 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:39.832525 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.833422 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:39.834675 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:39.835691 (+ 1016us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:39.835726 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:39.835754 (+ 28us) server_negotiation.cc:299] Negotiation successful
0503 08:13:39.836006 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20240503 08:13:39.836336 7959 tablet_service.cc:1467] Processing CreateTablet for tablet ee604fc8287e4ac3b8a9c3f8ad04066c (DEFAULT_TABLE table=test-table [id=06b0dc55c76948789727425770d294cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:39.836336 7818 tablet_service.cc:1467] Processing CreateTablet for tablet ee604fc8287e4ac3b8a9c3f8ad04066c (DEFAULT_TABLE table=test-table [id=06b0dc55c76948789727425770d294cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:39.836553 7678 tablet_service.cc:1467] Processing CreateTablet for tablet ee604fc8287e4ac3b8a9c3f8ad04066c (DEFAULT_TABLE table=test-table [id=06b0dc55c76948789727425770d294cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:39.836656 7959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee604fc8287e4ac3b8a9c3f8ad04066c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:39.836666 7818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee604fc8287e4ac3b8a9c3f8ad04066c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:39.836817 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee604fc8287e4ac3b8a9c3f8ad04066c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:39.839162 8049 tablet_bootstrap.cc:492] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: Bootstrap starting.
I20240503 08:13:39.839927 8051 tablet_bootstrap.cc:492] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: Bootstrap starting.
I20240503 08:13:39.840320 8049 tablet_bootstrap.cc:654] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:39.840643 8049 log.cc:824] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:39.840798 8051 tablet_bootstrap.cc:654] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:39.841099 8051 log.cc:824] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:39.841399 8050 tablet_bootstrap.cc:492] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: Bootstrap starting.
I20240503 08:13:39.841534 8049 tablet_bootstrap.cc:492] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: No bootstrap required, opened a new log
I20240503 08:13:39.841600 8049 ts_tablet_manager.cc:1367] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:39.841778 8051 tablet_bootstrap.cc:492] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: No bootstrap required, opened a new log
I20240503 08:13:39.841840 8051 ts_tablet_manager.cc:1367] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:39.842204 8050 tablet_bootstrap.cc:654] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:39.842507 8050 log.cc:824] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:39.843322 8050 tablet_bootstrap.cc:492] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: No bootstrap required, opened a new log
I20240503 08:13:39.843518 8050 ts_tablet_manager.cc:1367] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:39.843541 8049 raft_consensus.cc:348] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:39.843744 8049 raft_consensus.cc:374] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:39.843775 8049 raft_consensus.cc:729] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d96cc49c65b477c960e71873f824c57, State: Initialized, Role: FOLLOWER
I20240503 08:13:39.843700 8051 raft_consensus.cc:348] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:39.843856 8051 raft_consensus.cc:374] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:39.843887 8051 raft_consensus.cc:729] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06207493c36d45ed906a6d2f90721a6b, State: Initialized, Role: FOLLOWER
I20240503 08:13:39.843909 8049 consensus_queue.cc:260] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:39.843983 8051 consensus_queue.cc:260] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:39.844159 8049 ts_tablet_manager.cc:1398] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:39.844241 8051 ts_tablet_manager.cc:1398] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:39.844280 7744 heartbeater.cc:503] Master 127.6.68.254:40675 was elected leader, sending a full tablet report...
I20240503 08:13:39.844409 7885 heartbeater.cc:503] Master 127.6.68.254:40675 was elected leader, sending a full tablet report...
I20240503 08:13:39.845937 8050 raft_consensus.cc:348] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:39.846072 8050 raft_consensus.cc:374] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:39.846089 8050 raft_consensus.cc:729] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf174e6bc884f0cba6f65cb0c336e5e, State: Initialized, Role: FOLLOWER
W20240503 08:13:39.846179 7886 tablet.cc:2369] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:13:39.846174 8050 consensus_queue.cc:260] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:39.846422 8050 ts_tablet_manager.cc:1398] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:13:39.846478 8025 heartbeater.cc:503] Master 127.6.68.254:40675 was elected leader, sending a full tablet report...
W20240503 08:13:39.927304 7745 tablet.cc:2369] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:13:40.048076 8026 tablet.cc:2369] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:13:40.126271 8057 raft_consensus.cc:482] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:40.126442 8057 raft_consensus.cc:504] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.126824 8057 leader_election.cc:288] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06207493c36d45ed906a6d2f90721a6b (127.6.68.194:34989), 4d96cc49c65b477c960e71873f824c57 (127.6.68.193:39487)
I20240503 08:13:40.129354 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.127184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39487 (local address 127.6.68.195:42913)
0503 08:13:40.127358 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:40.127368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:40.127449 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:40.127615 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:40.127616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:40.127617 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:40.127889 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.127891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.128575 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.128576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:40.129119 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.129121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.129191 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.129195 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:13:40.129228 (+ 33us) client_negotiation.cc:240] Negotiation successful
0503 08:13:40.129263 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
I20240503 08:13:40.129338 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.127072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:34989 (local address 127.6.68.195:43097)
0503 08:13:40.127147 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:40.127158 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:40.127229 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:40.127493 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:40.127495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:40.127497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:40.127730 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.127732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.128487 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.128488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:40.129108 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.129110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.129176 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.129180 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:13:40.129214 (+ 34us) client_negotiation.cc:240] Negotiation successful
0503 08:13:40.129248 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20240503 08:13:40.129845 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.127296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:42913 (local address 127.6.68.193:39487)
0503 08:13:40.127342 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:40.127343 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:40.127474 (+ 131us) server_negotiation.cc:407] Connection header received
0503 08:13:40.127508 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:40.127509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:40.127518 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:40.127601 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:40.127994 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.128490 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.129205 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.129676 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.129685 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:40.129727 (+ 42us) server_negotiation.cc:299] Negotiation successful
0503 08:13:40.129763 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240503 08:13:40.129904 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.127147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:43097 (local address 127.6.68.194:34989)
0503 08:13:40.127213 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:40.127215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:40.127227 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:13:40.127372 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:40.127374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:40.127386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:40.127448 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:40.127806 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.128392 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.129192 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.129756 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.129767 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:40.129793 (+ 26us) server_negotiation.cc:299] Negotiation successful
0503 08:13:40.129828 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240503 08:13:40.130117 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee604fc8287e4ac3b8a9c3f8ad04066c" candidate_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d96cc49c65b477c960e71873f824c57" is_pre_election: true
I20240503 08:13:40.130160 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee604fc8287e4ac3b8a9c3f8ad04066c" candidate_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06207493c36d45ed906a6d2f90721a6b" is_pre_election: true
I20240503 08:13:40.130285 7698 raft_consensus.cc:2454] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf174e6bc884f0cba6f65cb0c336e5e in term 0.
I20240503 08:13:40.130285 7839 raft_consensus.cc:2454] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf174e6bc884f0cba6f65cb0c336e5e in term 0.
I20240503 08:13:40.130584 7911 leader_election.cc:302] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf174e6bc884f0cba6f65cb0c336e5e, 4d96cc49c65b477c960e71873f824c57; no voters:
I20240503 08:13:40.130720 8057 raft_consensus.cc:2781] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:40.130782 8057 raft_consensus.cc:482] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:40.130810 8057 raft_consensus.cc:3037] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:40.131527 8057 raft_consensus.cc:504] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.131695 8057 leader_election.cc:288] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [CANDIDATE]: Term 1 election: Requested vote from peers 06207493c36d45ed906a6d2f90721a6b (127.6.68.194:34989), 4d96cc49c65b477c960e71873f824c57 (127.6.68.193:39487)
I20240503 08:13:40.131923 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee604fc8287e4ac3b8a9c3f8ad04066c" candidate_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06207493c36d45ed906a6d2f90721a6b"
I20240503 08:13:40.131923 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee604fc8287e4ac3b8a9c3f8ad04066c" candidate_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d96cc49c65b477c960e71873f824c57"
I20240503 08:13:40.132007 7839 raft_consensus.cc:3037] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:40.132007 7698 raft_consensus.cc:3037] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:40.132862 7698 raft_consensus.cc:2454] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf174e6bc884f0cba6f65cb0c336e5e in term 1.
I20240503 08:13:40.132862 7839 raft_consensus.cc:2454] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf174e6bc884f0cba6f65cb0c336e5e in term 1.
I20240503 08:13:40.133054 7911 leader_election.cc:302] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06207493c36d45ed906a6d2f90721a6b, 3cf174e6bc884f0cba6f65cb0c336e5e; no voters:
I20240503 08:13:40.133176 8057 raft_consensus.cc:2781] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:40.133342 8057 raft_consensus.cc:686] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 LEADER]: Becoming Leader. State: Replica: 3cf174e6bc884f0cba6f65cb0c336e5e, State: Running, Role: LEADER
I20240503 08:13:40.133443 8057 consensus_queue.cc:237] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.134220 7539 catalog_manager.cc:5484] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cf174e6bc884f0cba6f65cb0c336e5e (127.6.68.195). New cstate: current_term: 1 leader_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.190364 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.188213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.6.68.195:35547)
0503 08:13:40.188295 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:40.188297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:40.188304 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:13:40.188344 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:40.188345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:40.188360 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:40.188426 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:40.188681 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.189193 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.189908 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.190050 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.190087 (+ 37us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:40.190175 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:40.190267 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:40.190280 (+ 13us) server_negotiation.cc:299] Negotiation successful
0503 08:13:40.190303 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20240503 08:13:40.192544 7839 raft_consensus.cc:1261] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b [term 1 FOLLOWER]: Refusing update from remote peer 3cf174e6bc884f0cba6f65cb0c336e5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:40.192544 7698 raft_consensus.cc:1261] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57 [term 1 FOLLOWER]: Refusing update from remote peer 3cf174e6bc884f0cba6f65cb0c336e5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:40.192885 8057 consensus_queue.cc:1035] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:40.193018 8059 consensus_queue.cc:1035] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:40.195295 8062 mvcc.cc:205] Tried to move back new op lower bound from 7023509586705973248 to 7023509586467147776. Current Snapshot: MvccSnapshot[applied={T|T < 7023509586705973248}]
I20240503 08:13:40.195538 8064 mvcc.cc:205] Tried to move back new op lower bound from 7023509586705973248 to 7023509586467147776. Current Snapshot: MvccSnapshot[applied={T|T < 7023509586705973248}]
I20240503 08:13:40.195605 8063 mvcc.cc:205] Tried to move back new op lower bound from 7023509586705973248 to 7023509586467147776. Current Snapshot: MvccSnapshot[applied={T|T < 7023509586705973248}]
I20240503 08:13:40.198346 7539 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36736:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:40.198463 7539 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36736:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:40.199178 7539 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139: Sending DeleteTablet for 3 replicas of tablet ee604fc8287e4ac3b8a9c3f8ad04066c
I20240503 08:13:40.199522 7818 tablet_service.cc:1514] Processing DeleteTablet for tablet ee604fc8287e4ac3b8a9c3f8ad04066c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:40 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:39904
I20240503 08:13:40.199584 7959 tablet_service.cc:1514] Processing DeleteTablet for tablet ee604fc8287e4ac3b8a9c3f8ad04066c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:40 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:42178
I20240503 08:13:40.199600 7678 tablet_service.cc:1514] Processing DeleteTablet for tablet ee604fc8287e4ac3b8a9c3f8ad04066c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:40 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:42550
I20240503 08:13:40.199898 8074 tablet_replica.cc:330] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: stopping tablet replica
I20240503 08:13:40.199924 8073 tablet_replica.cc:330] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: stopping tablet replica
I20240503 08:13:40.200006 8073 raft_consensus.cc:2229] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:40.200019 8074 raft_consensus.cc:2229] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:40.200112 8074 raft_consensus.cc:2258] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:40.200147 8073 raft_consensus.cc:2258] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:40.200126 7539 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:13:40.200273 7539 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:13:40.200331 8072 tablet_replica.cc:330] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: stopping tablet replica
I20240503 08:13:40.200410 8072 raft_consensus.cc:2229] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:40.200523 8072 raft_consensus.cc:2258] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:40.200816 8073 ts_tablet_manager.cc:1875] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:40.200938 8072 ts_tablet_manager.cc:1875] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:40.202160 7818 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb5c91df0654cf7953829aa705b7254 (DEFAULT_TABLE table=test-table [id=27551bda12c24594933f4df964504e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:40.202277 7818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb5c91df0654cf7953829aa705b7254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:40.202669 8074 ts_tablet_manager.cc:1875] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:40.202845 8072 ts_tablet_manager.cc:1888] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:13:40.202901 8072 log.cc:1196] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/ee604fc8287e4ac3b8a9c3f8ad04066c
I20240503 08:13:40.203169 8072 ts_tablet_manager.cc:1909] T ee604fc8287e4ac3b8a9c3f8ad04066c P 3cf174e6bc884f0cba6f65cb0c336e5e: Deleting consensus metadata
I20240503 08:13:40.203660 8051 tablet_bootstrap.cc:492] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b: Bootstrap starting.
I20240503 08:13:40.203922 7678 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb5c91df0654cf7953829aa705b7254 (DEFAULT_TABLE table=test-table [id=27551bda12c24594933f4df964504e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:40.204010 8051 tablet_bootstrap.cc:654] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:40.204044 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb5c91df0654cf7953829aa705b7254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:40.204209 8074 ts_tablet_manager.cc:1888] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:13:40.204262 8074 log.cc:1196] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/ee604fc8287e4ac3b8a9c3f8ad04066c
I20240503 08:13:40.204520 8074 ts_tablet_manager.cc:1909] T ee604fc8287e4ac3b8a9c3f8ad04066c P 06207493c36d45ed906a6d2f90721a6b: Deleting consensus metadata
I20240503 08:13:40.204602 8051 tablet_bootstrap.cc:492] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b: No bootstrap required, opened a new log
I20240503 08:13:40.204640 8051 ts_tablet_manager.cc:1367] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:40.204813 8073 ts_tablet_manager.cc:1888] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:13:40.204787 8051 raft_consensus.cc:348] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.204854 8051 raft_consensus.cc:374] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:40.204869 8073 log.cc:1196] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/ee604fc8287e4ac3b8a9c3f8ad04066c
I20240503 08:13:40.204875 8051 raft_consensus.cc:729] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06207493c36d45ed906a6d2f90721a6b, State: Initialized, Role: FOLLOWER
I20240503 08:13:40.204926 8051 consensus_queue.cc:260] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.205035 8051 ts_tablet_manager.cc:1398] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:40.205147 8073 ts_tablet_manager.cc:1909] T ee604fc8287e4ac3b8a9c3f8ad04066c P 4d96cc49c65b477c960e71873f824c57: Deleting consensus metadata
I20240503 08:13:40.205288 8049 tablet_bootstrap.cc:492] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57: Bootstrap starting.
I20240503 08:13:40.205670 8049 tablet_bootstrap.cc:654] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:40.205721 7525 catalog_manager.cc:4830] TS 4d96cc49c65b477c960e71873f824c57 (127.6.68.193:39487): tablet ee604fc8287e4ac3b8a9c3f8ad04066c (table test-table [id=06b0dc55c76948789727425770d294cc]) successfully deleted
I20240503 08:13:40.205817 7525 catalog_manager.cc:4830] TS 06207493c36d45ed906a6d2f90721a6b (127.6.68.194:34989): tablet ee604fc8287e4ac3b8a9c3f8ad04066c (table test-table [id=06b0dc55c76948789727425770d294cc]) successfully deleted
I20240503 08:13:40.205993 7525 catalog_manager.cc:4830] TS 3cf174e6bc884f0cba6f65cb0c336e5e (127.6.68.195:35547): tablet ee604fc8287e4ac3b8a9c3f8ad04066c (table test-table [id=06b0dc55c76948789727425770d294cc]) successfully deleted
I20240503 08:13:40.206046 7959 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb5c91df0654cf7953829aa705b7254 (DEFAULT_TABLE table=test-table [id=27551bda12c24594933f4df964504e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:40.206146 7959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb5c91df0654cf7953829aa705b7254. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:40.206326 8049 tablet_bootstrap.cc:492] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57: No bootstrap required, opened a new log
I20240503 08:13:40.206370 8049 ts_tablet_manager.cc:1367] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:40.206502 8049 raft_consensus.cc:348] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.206552 8049 raft_consensus.cc:374] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:40.206574 8049 raft_consensus.cc:729] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d96cc49c65b477c960e71873f824c57, State: Initialized, Role: FOLLOWER
I20240503 08:13:40.206625 8049 consensus_queue.cc:260] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.206723 8049 ts_tablet_manager.cc:1398] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:40.207413 8050 tablet_bootstrap.cc:492] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e: Bootstrap starting.
I20240503 08:13:40.207815 8050 tablet_bootstrap.cc:654] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:40.208400 8050 tablet_bootstrap.cc:492] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e: No bootstrap required, opened a new log
I20240503 08:13:40.208447 8050 ts_tablet_manager.cc:1367] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:40.208597 8050 raft_consensus.cc:348] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.208662 8050 raft_consensus.cc:374] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:40.208680 8050 raft_consensus.cc:729] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf174e6bc884f0cba6f65cb0c336e5e, State: Initialized, Role: FOLLOWER
I20240503 08:13:40.208738 8050 consensus_queue.cc:260] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.208832 8050 ts_tablet_manager.cc:1398] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:40.600852 8067 raft_consensus.cc:482] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:40.600979 8067 raft_consensus.cc:504] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.601158 8067 leader_election.cc:288] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d96cc49c65b477c960e71873f824c57 (127.6.68.193:39487), 06207493c36d45ed906a6d2f90721a6b (127.6.68.194:34989)
I20240503 08:13:40.601405 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb5c91df0654cf7953829aa705b7254" candidate_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d96cc49c65b477c960e71873f824c57" is_pre_election: true
I20240503 08:13:40.601411 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb5c91df0654cf7953829aa705b7254" candidate_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06207493c36d45ed906a6d2f90721a6b" is_pre_election: true
I20240503 08:13:40.601507 7698 raft_consensus.cc:2454] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf174e6bc884f0cba6f65cb0c336e5e in term 0.
I20240503 08:13:40.601508 7839 raft_consensus.cc:2454] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf174e6bc884f0cba6f65cb0c336e5e in term 0.
I20240503 08:13:40.601683 7911 leader_election.cc:302] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06207493c36d45ed906a6d2f90721a6b, 3cf174e6bc884f0cba6f65cb0c336e5e; no voters:
I20240503 08:13:40.601790 8067 raft_consensus.cc:2781] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:40.601835 8067 raft_consensus.cc:482] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:40.601850 8067 raft_consensus.cc:3037] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:40.602561 8067 raft_consensus.cc:504] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.602710 8067 leader_election.cc:288] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [CANDIDATE]: Term 1 election: Requested vote from peers 4d96cc49c65b477c960e71873f824c57 (127.6.68.193:39487), 06207493c36d45ed906a6d2f90721a6b (127.6.68.194:34989)
I20240503 08:13:40.602906 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb5c91df0654cf7953829aa705b7254" candidate_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d96cc49c65b477c960e71873f824c57"
I20240503 08:13:40.602909 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb5c91df0654cf7953829aa705b7254" candidate_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06207493c36d45ed906a6d2f90721a6b"
I20240503 08:13:40.602975 7698 raft_consensus.cc:3037] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:40.602977 7839 raft_consensus.cc:3037] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:40.603565 7698 raft_consensus.cc:2454] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf174e6bc884f0cba6f65cb0c336e5e in term 1.
I20240503 08:13:40.603650 7839 raft_consensus.cc:2454] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf174e6bc884f0cba6f65cb0c336e5e in term 1.
I20240503 08:13:40.603786 7911 leader_election.cc:302] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf174e6bc884f0cba6f65cb0c336e5e, 4d96cc49c65b477c960e71873f824c57; no voters:
I20240503 08:13:40.603907 8067 raft_consensus.cc:2781] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:40.603973 8067 raft_consensus.cc:686] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 LEADER]: Becoming Leader. State: Replica: 3cf174e6bc884f0cba6f65cb0c336e5e, State: Running, Role: LEADER
I20240503 08:13:40.604030 8067 consensus_queue.cc:237] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.604681 7539 catalog_manager.cc:5484] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cf174e6bc884f0cba6f65cb0c336e5e (127.6.68.195). New cstate: current_term: 1 leader_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } health_report { overall_health: HEALTHY } } }
May 03 08:13:40 dist-test-slave-3g67 krb5kdc[7495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:40.689433 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.676503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40675 (local address 127.0.0.1:36738)
0503 08:13:40.676682 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:40.676696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:40.676779 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:40.676950 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:40.676951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:40.677301 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:40.677468 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.677470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.678128 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.678129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:40.678479 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.678481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.678550 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.685245 (+ 6695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:40.685250 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:40.686831 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:40.688526 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:40.688534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:40.688537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:40.688742 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:40.688878 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:40.688878 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:40.688878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:40.688904 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:40.689152 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:40.689156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:40.689200 (+ 44us) client_negotiation.cc:769] Sending connection context
0503 08:13:40.689250 (+ 50us) client_negotiation.cc:240] Negotiation successful
0503 08:13:40.689355 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20240503 08:13:40.689523 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.676581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.6.68.254:40675)
0503 08:13:40.676718 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:40.676720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:40.676754 (+ 34us) server_negotiation.cc:407] Connection header received
0503 08:13:40.676830 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:40.676830 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:40.676839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:40.676905 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:40.677537 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.678039 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.678555 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.678719 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.686942 (+ 8223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:40.686949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:40.686949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:40.686955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:40.688438 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:40.688810 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:40.688811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:40.688811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:40.688828 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:40.688988 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:40.688989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:40.688990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:40.689094 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:40.689160 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:40.689301 (+ 141us) server_negotiation.cc:299] Negotiation successful
0503 08:13:40.689420 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20240503 08:13:40.691506 7539 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:36738:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240503 08:13:40.691736 7539 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:13:40.693280 7678 tablet_service.cc:1467] Processing CreateTablet for tablet f2ed37edfc554c01b88e9f3c2bff8c46 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08503b18363a4a08866262437f7a2ee2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240503 08:13:40.693318 7818 tablet_service.cc:1467] Processing CreateTablet for tablet f2ed37edfc554c01b88e9f3c2bff8c46 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08503b18363a4a08866262437f7a2ee2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240503 08:13:40.693393 7959 tablet_service.cc:1467] Processing CreateTablet for tablet f2ed37edfc554c01b88e9f3c2bff8c46 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08503b18363a4a08866262437f7a2ee2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240503 08:13:40.693459 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ed37edfc554c01b88e9f3c2bff8c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:40.693465 7818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ed37edfc554c01b88e9f3c2bff8c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:40.693526 7959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ed37edfc554c01b88e9f3c2bff8c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:40.694545 8051 tablet_bootstrap.cc:492] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: Bootstrap starting.
I20240503 08:13:40.694888 8051 tablet_bootstrap.cc:654] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:40.694941 8050 tablet_bootstrap.cc:492] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e: Bootstrap starting.
I20240503 08:13:40.694959 8049 tablet_bootstrap.cc:492] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57: Bootstrap starting.
I20240503 08:13:40.695261 8049 tablet_bootstrap.cc:654] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:40.695315 8050 tablet_bootstrap.cc:654] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:40.695835 8051 tablet_bootstrap.cc:492] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: No bootstrap required, opened a new log
I20240503 08:13:40.695890 8051 ts_tablet_manager.cc:1367] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:40.696003 8049 tablet_bootstrap.cc:492] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57: No bootstrap required, opened a new log
I20240503 08:13:40.696066 8049 ts_tablet_manager.cc:1367] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:40.696040 8051 raft_consensus.cc:348] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.696094 8050 tablet_bootstrap.cc:492] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e: No bootstrap required, opened a new log
I20240503 08:13:40.696110 8051 raft_consensus.cc:374] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:40.696130 8051 raft_consensus.cc:729] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06207493c36d45ed906a6d2f90721a6b, State: Initialized, Role: FOLLOWER
I20240503 08:13:40.696136 8050 ts_tablet_manager.cc:1367] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:40.696192 8051 consensus_queue.cc:260] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.696221 8049 raft_consensus.cc:348] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.696285 8049 raft_consensus.cc:374] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:40.696288 8051 ts_tablet_manager.cc:1398] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:40.696313 8049 raft_consensus.cc:729] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d96cc49c65b477c960e71873f824c57, State: Initialized, Role: FOLLOWER
I20240503 08:13:40.696306 8050 raft_consensus.cc:348] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.696332 8056 tablet_replica.cc:439] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } } }
I20240503 08:13:40.696377 8050 raft_consensus.cc:374] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:40.696362 8049 consensus_queue.cc:260] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.696398 8050 raft_consensus.cc:729] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf174e6bc884f0cba6f65cb0c336e5e, State: Initialized, Role: FOLLOWER
I20240503 08:13:40.696401 8056 tablet_replica.cc:442] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:40.696440 8050 consensus_queue.cc:260] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.696466 8049 ts_tablet_manager.cc:1398] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:40.696525 8050 ts_tablet_manager.cc:1398] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:13:40.696626 8055 tablet_replica.cc:439] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } } }
I20240503 08:13:40.696621 8059 tablet_replica.cc:439] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } } }
I20240503 08:13:40.696692 8059 tablet_replica.cc:442] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:40.696688 8055 tablet_replica.cc:442] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:40.714125 8056 raft_consensus.cc:482] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:40.714229 8056 raft_consensus.cc:504] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.714548 8056 leader_election.cc:288] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d96cc49c65b477c960e71873f824c57 (127.6.68.193:39487), 3cf174e6bc884f0cba6f65cb0c336e5e (127.6.68.195:35547)
I20240503 08:13:40.717093 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.714718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39487 (local address 127.6.68.194:57319)
0503 08:13:40.714861 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:40.714874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:40.714951 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:40.715169 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:40.715170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:40.715173 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:40.715420 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.715424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.716196 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.716197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:40.716823 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.716825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.716898 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.716903 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:13:40.716938 (+ 35us) client_negotiation.cc:240] Negotiation successful
0503 08:13:40.716981 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20240503 08:13:40.717095 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.714873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:35547 (local address 127.6.68.194:42981)
0503 08:13:40.715012 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:40.715021 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:40.715065 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:40.715241 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:40.715242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:40.715245 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:40.715479 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.715481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.716207 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.716208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:40.716825 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.716826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.716901 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.716906 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:13:40.716942 (+ 36us) client_negotiation.cc:240] Negotiation successful
0503 08:13:40.716983 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":42,"threads_started":1}
I20240503 08:13:40.717543 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.714807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:57319 (local address 127.6.68.193:39487)
0503 08:13:40.714951 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:40.714953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:40.714964 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:13:40.715058 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:40.715059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:40.715078 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:40.715158 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:40.715553 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.716107 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.716919 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.717380 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.717388 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:40.717432 (+ 44us) server_negotiation.cc:299] Negotiation successful
0503 08:13:40.717470 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20240503 08:13:40.717641 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.714955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:42981 (local address 127.6.68.195:35547)
0503 08:13:40.715073 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:40.715075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:40.715084 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:13:40.715111 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:40.715112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:40.715122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:40.715181 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:40.715562 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.716111 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.716924 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.717511 (+ 587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.717518 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:40.717542 (+ 24us) server_negotiation.cc:299] Negotiation successful
0503 08:13:40.717574 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20240503 08:13:40.717748 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ed37edfc554c01b88e9f3c2bff8c46" candidate_uuid: "06207493c36d45ed906a6d2f90721a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d96cc49c65b477c960e71873f824c57" is_pre_election: true
I20240503 08:13:40.717834 7698 raft_consensus.cc:2454] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06207493c36d45ed906a6d2f90721a6b in term 0.
I20240503 08:13:40.717898 7979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ed37edfc554c01b88e9f3c2bff8c46" candidate_uuid: "06207493c36d45ed906a6d2f90721a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" is_pre_election: true
I20240503 08:13:40.717995 7979 raft_consensus.cc:2454] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06207493c36d45ed906a6d2f90721a6b in term 0.
I20240503 08:13:40.718021 7770 leader_election.cc:302] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06207493c36d45ed906a6d2f90721a6b, 4d96cc49c65b477c960e71873f824c57; no voters:
I20240503 08:13:40.718210 8056 raft_consensus.cc:2781] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:40.718282 8056 raft_consensus.cc:482] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:40.718307 8056 raft_consensus.cc:3037] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:40.718997 8056 raft_consensus.cc:504] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.719127 8056 leader_election.cc:288] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [CANDIDATE]: Term 1 election: Requested vote from peers 4d96cc49c65b477c960e71873f824c57 (127.6.68.193:39487), 3cf174e6bc884f0cba6f65cb0c336e5e (127.6.68.195:35547)
I20240503 08:13:40.719357 7979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ed37edfc554c01b88e9f3c2bff8c46" candidate_uuid: "06207493c36d45ed906a6d2f90721a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e"
I20240503 08:13:40.719385 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ed37edfc554c01b88e9f3c2bff8c46" candidate_uuid: "06207493c36d45ed906a6d2f90721a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d96cc49c65b477c960e71873f824c57"
I20240503 08:13:40.719441 7979 raft_consensus.cc:3037] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:40.719445 7698 raft_consensus.cc:3037] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:40.720244 7698 raft_consensus.cc:2454] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06207493c36d45ed906a6d2f90721a6b in term 1.
I20240503 08:13:40.720324 7979 raft_consensus.cc:2454] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06207493c36d45ed906a6d2f90721a6b in term 1.
I20240503 08:13:40.720445 7770 leader_election.cc:302] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06207493c36d45ed906a6d2f90721a6b, 4d96cc49c65b477c960e71873f824c57; no voters:
I20240503 08:13:40.720537 8056 raft_consensus.cc:2781] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:40.720659 8056 raft_consensus.cc:686] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [term 1 LEADER]: Becoming Leader. State: Replica: 06207493c36d45ed906a6d2f90721a6b, State: Running, Role: LEADER
I20240503 08:13:40.720724 8056 consensus_queue.cc:237] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } }
I20240503 08:13:40.721071 8093 tablet_replica.cc:439] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: TxnStatusTablet state changed. Reason: New leader 06207493c36d45ed906a6d2f90721a6b. Latest consensus state: current_term: 1 leader_uuid: "06207493c36d45ed906a6d2f90721a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } } }
I20240503 08:13:40.721158 8093 tablet_replica.cc:442] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: This TxnStatusTablet replica's current role is: LEADER
I20240503 08:13:40.721415 8095 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240503 08:13:40.721510 8095 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240503 08:13:40.721524 7539 catalog_manager.cc:5484] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 06207493c36d45ed906a6d2f90721a6b (127.6.68.194). New cstate: current_term: 1 leader_uuid: "06207493c36d45ed906a6d2f90721a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.734658 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.731968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:34989 (local address 127.0.0.1:39916)
0503 08:13:40.732053 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:40.732065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:40.732120 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:40.732410 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:40.732410 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:40.732412 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0503 08:13:40.732561 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.732562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.733559 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.733559 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:40.733971 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.733972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.734027 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.734045 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0503 08:13:40.734467 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0503 08:13:40.734472 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:13:40.734525 (+ 53us) client_negotiation.cc:240] Negotiation successful
0503 08:13:40.734575 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20240503 08:13:40.734746 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.732062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.6.68.194:34989)
0503 08:13:40.732229 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:40.732232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:40.732253 (+ 21us) server_negotiation.cc:407] Connection header received
0503 08:13:40.732293 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:40.732293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:40.732315 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:40.732412 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:40.732624 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.733476 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.734048 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.734235 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.734281 (+ 46us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:40.734387 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:40.734474 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:40.734577 (+ 103us) server_negotiation.cc:299] Negotiation successful
0503 08:13:40.734604 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20240503 08:13:40.736442 7698 raft_consensus.cc:1261] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57 [term 1 FOLLOWER]: Refusing update from remote peer 06207493c36d45ed906a6d2f90721a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:40.736867 8093 consensus_queue.cc:1035] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:40.736559 7979 raft_consensus.cc:1261] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 FOLLOWER]: Refusing update from remote peer 06207493c36d45ed906a6d2f90721a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:40.737354 8093 consensus_queue.cc:1035] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:40.737696 8055 tablet_replica.cc:439] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57: TxnStatusTablet state changed. Reason: New leader 06207493c36d45ed906a6d2f90721a6b. Latest consensus state: current_term: 1 leader_uuid: "06207493c36d45ed906a6d2f90721a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } } }
I20240503 08:13:40.737779 8055 tablet_replica.cc:442] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:40.738114 8059 tablet_replica.cc:439] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e: TxnStatusTablet state changed. Reason: New leader 06207493c36d45ed906a6d2f90721a6b. Latest consensus state: current_term: 1 leader_uuid: "06207493c36d45ed906a6d2f90721a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } } }
I20240503 08:13:40.738205 8059 tablet_replica.cc:442] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:40.739128 8056 tablet_replica.cc:439] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06207493c36d45ed906a6d2f90721a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } } }
I20240503 08:13:40.739223 8056 tablet_replica.cc:442] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: This TxnStatusTablet replica's current role is: LEADER
I20240503 08:13:40.739163 8104 tablet_replica.cc:439] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06207493c36d45ed906a6d2f90721a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } } }
I20240503 08:13:40.739286 8104 tablet_replica.cc:442] T f2ed37edfc554c01b88e9f3c2bff8c46 P 06207493c36d45ed906a6d2f90721a6b: This TxnStatusTablet replica's current role is: LEADER
I20240503 08:13:40.739650 8055 tablet_replica.cc:439] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06207493c36d45ed906a6d2f90721a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } } }
I20240503 08:13:40.739732 8055 tablet_replica.cc:442] T f2ed37edfc554c01b88e9f3c2bff8c46 P 4d96cc49c65b477c960e71873f824c57: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:40.740082 8059 tablet_replica.cc:439] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06207493c36d45ed906a6d2f90721a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 } } peers { permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 } } peers { permanent_uuid: "3cf174e6bc884f0cba6f65cb0c336e5e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 35547 } } }
I20240503 08:13:40.740173 8059 tablet_replica.cc:442] T f2ed37edfc554c01b88e9f3c2bff8c46 P 3cf174e6bc884f0cba6f65cb0c336e5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:40.745555 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.743100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:34989 (local address 127.6.68.195:34143)
0503 08:13:40.743241 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:40.743251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:40.743310 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:40.743446 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:40.743447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:40.743449 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0503 08:13:40.743613 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.743615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.744607 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.744608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:40.745020 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.745021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.745077 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.745093 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0503 08:13:40.745421 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0503 08:13:40.745422 (+ 1us) client_negotiation.cc:769] Sending connection context
0503 08:13:40.745457 (+ 35us) client_negotiation.cc:240] Negotiation successful
0503 08:13:40.745491 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20240503 08:13:40.745575 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.743181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:34143 (local address 127.6.68.194:34989)
0503 08:13:40.743223 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:40.743225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:40.743312 (+ 87us) server_negotiation.cc:407] Connection header received
0503 08:13:40.743354 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:40.743354 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:40.743365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:40.743433 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:40.743691 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.744515 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.745094 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.745303 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.745325 (+ 22us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:40.745355 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:40.745416 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:40.745475 (+ 59us) server_negotiation.cc:299] Negotiation successful
0503 08:13:40.745498 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:13:40.747860 7698 raft_consensus.cc:1261] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57 [term 1 FOLLOWER]: Refusing update from remote peer 3cf174e6bc884f0cba6f65cb0c336e5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:40.747860 7839 raft_consensus.cc:1261] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b [term 1 FOLLOWER]: Refusing update from remote peer 3cf174e6bc884f0cba6f65cb0c336e5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:40.748101 8067 consensus_queue.cc:1035] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d96cc49c65b477c960e71873f824c57" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:40.748237 8067 consensus_queue.cc:1035] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06207493c36d45ed906a6d2f90721a6b" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:40.757511 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.754716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:35547 (local address 127.6.68.194:52903)
0503 08:13:40.754974 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:40.754983 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:40.755030 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:40.755151 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:40.755152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:40.755153 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0503 08:13:40.755349 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.755350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.756325 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.756326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:40.756934 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:40.756937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.757002 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.757024 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0503 08:13:40.757372 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0503 08:13:40.757373 (+ 1us) client_negotiation.cc:769] Sending connection context
0503 08:13:40.757405 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:13:40.757439 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":74,"threads_started":1}
I20240503 08:13:40.757516 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.754772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:52903 (local address 127.6.68.195:35547)
0503 08:13:40.754818 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:40.754820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:40.755022 (+ 202us) server_negotiation.cc:407] Connection header received
0503 08:13:40.755072 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:40.755073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:40.755084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:40.755150 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:40.755417 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.756199 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.757047 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.757235 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.757265 (+ 30us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:40.757298 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:40.757355 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:40.757409 (+ 54us) server_negotiation.cc:299] Negotiation successful
0503 08:13:40.757436 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:13:40.766678 7535 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36736:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:40.766814 7535 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36736:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:40.767524 7535 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 801c42b2699945be950cf3da7cf0b139: Sending DeleteTablet for 3 replicas of tablet 9eb5c91df0654cf7953829aa705b7254
I20240503 08:13:40.767807 7678 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eb5c91df0654cf7953829aa705b7254 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:40 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:42550
I20240503 08:13:40.767926 7818 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eb5c91df0654cf7953829aa705b7254 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:40 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:39904
I20240503 08:13:40.768103 7959 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eb5c91df0654cf7953829aa705b7254 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:40 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:42178
I20240503 08:13:40.768280 8120 tablet_replica.cc:330] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e: stopping tablet replica
I20240503 08:13:40.768334 8120 raft_consensus.cc:2229] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:40.768409 8120 raft_consensus.cc:2258] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:40.768607 8117 tablet_replica.cc:330] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57: stopping tablet replica
I20240503 08:13:40.768626 8118 tablet_replica.cc:330] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b: stopping tablet replica
I20240503 08:13:40.768675 8117 raft_consensus.cc:2229] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:40.768729 8117 raft_consensus.cc:2258] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:40.768740 8118 raft_consensus.cc:2229] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:40.768857 8118 raft_consensus.cc:2258] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:40.769101 8120 ts_tablet_manager.cc:1875] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:40.769439 8117 ts_tablet_manager.cc:1875] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:40.769213 8118 ts_tablet_manager.cc:1875] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:40.771922 8118 ts_tablet_manager.cc:1888] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:13:40.771983 8118 log.cc:1196] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/9eb5c91df0654cf7953829aa705b7254
I20240503 08:13:40.772209 8118 ts_tablet_manager.cc:1909] T 9eb5c91df0654cf7953829aa705b7254 P 06207493c36d45ed906a6d2f90721a6b: Deleting consensus metadata
I20240503 08:13:40.772837 8117 ts_tablet_manager.cc:1888] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:13:40.772912 8117 log.cc:1196] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/9eb5c91df0654cf7953829aa705b7254
I20240503 08:13:40.773136 8117 ts_tablet_manager.cc:1909] T 9eb5c91df0654cf7953829aa705b7254 P 4d96cc49c65b477c960e71873f824c57: Deleting consensus metadata
I20240503 08:13:40.773126 7525 catalog_manager.cc:4830] TS 06207493c36d45ed906a6d2f90721a6b (127.6.68.194:34989): tablet 9eb5c91df0654cf7953829aa705b7254 (table test-table [id=27551bda12c24594933f4df964504e3f]) successfully deleted
I20240503 08:13:40.773345 8120 ts_tablet_manager.cc:1888] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:13:40.773414 8120 log.cc:1196] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/9eb5c91df0654cf7953829aa705b7254
I20240503 08:13:40.773695 8120 ts_tablet_manager.cc:1909] T 9eb5c91df0654cf7953829aa705b7254 P 3cf174e6bc884f0cba6f65cb0c336e5e: Deleting consensus metadata
I20240503 08:13:40.774025 7525 catalog_manager.cc:4830] TS 3cf174e6bc884f0cba6f65cb0c336e5e (127.6.68.195:35547): tablet 9eb5c91df0654cf7953829aa705b7254 (table test-table [id=27551bda12c24594933f4df964504e3f]) successfully deleted
I20240503 08:13:40.774217 7525 catalog_manager.cc:4830] TS 4d96cc49c65b477c960e71873f824c57 (127.6.68.193:39487): tablet 9eb5c91df0654cf7953829aa705b7254 (table test-table [id=27551bda12c24594933f4df964504e3f]) successfully deleted
May 03 08:13:40 dist-test-slave-3g67 krb5kdc[7495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.68.193@KRBTEST.COM
I20240503 08:13:40.778003 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.768510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.6.68.193:39487)
0503 08:13:40.768566 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:40.768568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:40.769075 (+ 507us) server_negotiation.cc:407] Connection header received
0503 08:13:40.769223 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:40.769224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:40.769235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:40.769265 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:40.770284 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.770977 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.771836 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.772024 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.775269 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:40.775281 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:40.775281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:40.775288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:40.776997 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:40.777316 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:40.777319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:40.777319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:40.777349 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:40.777489 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:40.777489 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:40.777490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:40.777593 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:40.777729 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:40.777789 (+ 60us) server_negotiation.cc:299] Negotiation successful
0503 08:13:40.777895 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20240503 08:13:40.778229 7738 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42566
I20240503 08:13:40.784919 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:40.779163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.6.68.254:40675)
0503 08:13:40.779226 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:40.779228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:40.779283 (+ 55us) server_negotiation.cc:407] Connection header received
0503 08:13:40.779316 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:40.779316 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:40.779331 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:40.779400 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:40.779989 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.780828 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:40.781358 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:40.781537 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:40.782445 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:40.782449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:40.782449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:40.782456 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:40.784027 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:40.784311 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:40.784312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:40.784312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:40.784336 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:40.784451 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:40.784452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:40.784452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:40.784558 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:40.784659 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:40.784720 (+ 61us) server_negotiation.cc:299] Negotiation successful
0503 08:13:40.784814 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20240503 08:13:40.785163 7535 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36750
I20240503 08:13:40.785646 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 7613
I20240503 08:13:40.792484 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 7753
I20240503 08:13:40.797850 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 7894
I20240503 08:13:40.804056 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 7508
2024-05-03T08:13:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2788 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:13:40 dist-test-slave-3g67 krb5kdc[8126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:13:40 dist-test-slave-3g67 krb5kdc[8126](info): set up 2 sockets
May 03 08:13:40 dist-test-slave-3g67 krb5kdc[8126](info): commencing operation
krb5kdc: starting...
W20240503 08:13:41.760046 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:13:41 dist-test-slave-3g67 krb5kdc[8126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:13:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:13:41.879587 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34345
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:34345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:41.964454 8139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:41.964675 8139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:41.964699 8139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:41.966073 8139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:41.966122 8139 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240503 08:13:41.966136 8139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:41.966148 8139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:41.966161 8139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:41.966172 8139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:41.967891 8139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:34345
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:41.968108 8139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:41.968266 8139 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:41.973536 8144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:41.974095 8147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:41.974867 8145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:41.975677 8139 server_base.cc:1034] running on GCE node
I20240503 08:13:41.976032 8139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:41.976321 8139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:41.977480 8139 hybrid_clock.cc:648] HybridClock initialized: now 1714724021977459 us; error 37 us; skew 500 ppm
May 03 08:13:41 dist-test-slave-3g67 krb5kdc[8126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724021, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:41.980221 8139 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:13:41.980695 8139 webserver.cc:460] Webserver started at http://127.6.68.254:38519/ using document root <none> and password file <none>
I20240503 08:13:41.980885 8139 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:41.980949 8139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:41.981062 8139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:41.982105 8139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "690762e09f4b4777bc27f56ef249a079"
format_stamp: "Formatted at 2024-05-03 08:13:41 on dist-test-slave-3g67"
server_key: "e93ab763913b3a3bb94d53977759e2a8"
server_key_iv: "f110d3bf541efb4bd49c0da8c97c4bb6"
server_key_version: "encryptionkey@0"
I20240503 08:13:41.982416 8139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "690762e09f4b4777bc27f56ef249a079"
format_stamp: "Formatted at 2024-05-03 08:13:41 on dist-test-slave-3g67"
server_key: "e93ab763913b3a3bb94d53977759e2a8"
server_key_iv: "f110d3bf541efb4bd49c0da8c97c4bb6"
server_key_version: "encryptionkey@0"
I20240503 08:13:41.983711 8139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:13:41.984437 8154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:41.984606 8139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:41.984688 8139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "690762e09f4b4777bc27f56ef249a079"
format_stamp: "Formatted at 2024-05-03 08:13:41 on dist-test-slave-3g67"
server_key: "e93ab763913b3a3bb94d53977759e2a8"
server_key_iv: "f110d3bf541efb4bd49c0da8c97c4bb6"
server_key_version: "encryptionkey@0"
I20240503 08:13:41.984753 8139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:41.993685 8139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:41.996145 8139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:41.996320 8139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:42.000823 8139 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:34345
I20240503 08:13:42.000881 8217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:34345 every 8 connection(s)
I20240503 08:13:42.001219 8139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:13:42.001870 8218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:42.004128 8218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079: Bootstrap starting.
I20240503 08:13:42.004386 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 8139
I20240503 08:13:42.004473 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:13:42.004642 6419 external_mini_cluster.cc:1428] Setting key c3109d49bb111011936779bd5d73c882
I20240503 08:13:42.004770 8218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:42.005046 8218 log.cc:824] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:42.005761 8218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079: No bootstrap required, opened a new log
I20240503 08:13:42.007407 8218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690762e09f4b4777bc27f56ef249a079" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34345 } }
I20240503 08:13:42.007535 8218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:42.007565 8218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 690762e09f4b4777bc27f56ef249a079, State: Initialized, Role: FOLLOWER
I20240503 08:13:42.007700 8218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690762e09f4b4777bc27f56ef249a079" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34345 } }
I20240503 08:13:42.007778 8218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:42.007833 8218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:42.007879 8218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:42.008603 8218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690762e09f4b4777bc27f56ef249a079" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34345 } }
I20240503 08:13:42.008726 8218 leader_election.cc:302] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 690762e09f4b4777bc27f56ef249a079; no voters:
I20240503 08:13:42.008950 8218 leader_election.cc:288] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:42.009002 8223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:42.009222 8223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [term 1 LEADER]: Becoming Leader. State: Replica: 690762e09f4b4777bc27f56ef249a079, State: Running, Role: LEADER
I20240503 08:13:42.009238 8218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:42.009428 8223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690762e09f4b4777bc27f56ef249a079" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34345 } }
I20240503 08:13:42.009850 8224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "690762e09f4b4777bc27f56ef249a079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690762e09f4b4777bc27f56ef249a079" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34345 } } }
I20240503 08:13:42.009918 8224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:42.010149 8225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 690762e09f4b4777bc27f56ef249a079. Latest consensus state: current_term: 1 leader_uuid: "690762e09f4b4777bc27f56ef249a079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690762e09f4b4777bc27f56ef249a079" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34345 } } }
I20240503 08:13:42.010278 8225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:42.010614 8227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
May 03 08:13:42 dist-test-slave-3g67 krb5kdc[8126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:42.011153 8227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:42.012777 8227 catalog_manager.cc:1347] Generated new cluster ID: 933206b4931a4e9aa80699dcc85aaf66
I20240503 08:13:42.012823 8227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:13:42.015393 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.005143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.6.68.254:34345)
0503 08:13:42.005317 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.005327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.005344 (+ 17us) server_negotiation.cc:407] Connection header received
0503 08:13:42.005813 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.005821 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.005877 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.006040 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:42.006889 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.007716 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.008519 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.008704 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.011705 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:42.011723 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:42.011723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:42.011747 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:42.013966 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.014387 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.014390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.014390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.014440 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.014594 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.014595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.014595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.014808 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:42.014932 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.015076 (+ 144us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.015197 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20240503 08:13:42.031365 8227 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:42.032078 8227 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:42.037950 8227 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079: Generated new TSK 0
I20240503 08:13:42.038162 8227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:13:42.070945 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34345
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:42.153194 8244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:42.153406 8244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:42.153432 8244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:42.154798 8244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:42.154863 8244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:42.154899 8244 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240503 08:13:42.154920 8244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240503 08:13:42.154954 8244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:13:42.156661 8244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.68.254:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:42.156884 8244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:42.157089 8244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:42.162218 8249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:42.162683 8250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:42.163416 8252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:42.164314 8244 server_base.cc:1034] running on GCE node
I20240503 08:13:42.164520 8244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:42.164728 8244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:42.165863 8244 hybrid_clock.cc:648] HybridClock initialized: now 1714724022165846 us; error 39 us; skew 500 ppm
May 03 08:13:42 dist-test-slave-3g67 krb5kdc[8126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:42.168439 8244 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:13:42.168876 8244 webserver.cc:460] Webserver started at http://127.6.68.193:45535/ using document root <none> and password file <none>
I20240503 08:13:42.169057 8244 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:42.169101 8244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:42.169225 8244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:42.170255 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "17b6072e21c84f568a1d5fc931e1884f"
format_stamp: "Formatted at 2024-05-03 08:13:42 on dist-test-slave-3g67"
server_key: "735cd1fc909680e745c694b91e924b57"
server_key_iv: "aae40b6bc3982755629aa10c5fe6cca4"
server_key_version: "encryptionkey@0"
I20240503 08:13:42.170583 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "17b6072e21c84f568a1d5fc931e1884f"
format_stamp: "Formatted at 2024-05-03 08:13:42 on dist-test-slave-3g67"
server_key: "735cd1fc909680e745c694b91e924b57"
server_key_iv: "aae40b6bc3982755629aa10c5fe6cca4"
server_key_version: "encryptionkey@0"
I20240503 08:13:42.171875 8244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:13:42.172590 8259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:42.172773 8244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:42.172849 8244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "17b6072e21c84f568a1d5fc931e1884f"
format_stamp: "Formatted at 2024-05-03 08:13:42 on dist-test-slave-3g67"
server_key: "735cd1fc909680e745c694b91e924b57"
server_key_iv: "aae40b6bc3982755629aa10c5fe6cca4"
server_key_version: "encryptionkey@0"
I20240503 08:13:42.172914 8244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:42.184239 8244 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:42.186206 8244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:42.186336 8244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:42.186908 8244 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:42.186954 8244 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:42.186986 8244 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:42.187008 8244 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 03 08:13:42 dist-test-slave-3g67 krb5kdc[8126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:42.194044 8244 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:39791
I20240503 08:13:42.194391 8244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:13:42.194412 8374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:39791 every 8 connection(s)
I20240503 08:13:42.195561 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 8244
I20240503 08:13:42.195695 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:13:42.195896 6419 external_mini_cluster.cc:1428] Setting key 5976fbd6babcaacd6fecbe9334b8617d
I20240503 08:13:42.196439 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.187284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:33353 (local address 127.6.68.254:34345)
0503 08:13:42.187356 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.187359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.187566 (+ 207us) server_negotiation.cc:407] Connection header received
0503 08:13:42.188129 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.188131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.188147 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.188223 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:42.189382 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.189908 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.190506 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.190684 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.193307 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:42.193312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:42.193313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:42.193321 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:42.194989 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.195393 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.195394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.195395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.195419 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.195604 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.195605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.195605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.195765 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:42.195835 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.196205 (+ 370us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.196341 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240503 08:13:42.196465 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.187285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34345 (local address 127.6.68.193:33353)
0503 08:13:42.187442 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.187471 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.188022 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.188290 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.188293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.188712 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:42.189260 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.189268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.190011 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.190012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.190421 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.190423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.190565 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.191260 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:42.191287 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:42.193199 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:42.195084 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.195089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.195092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.195322 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.195486 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.195487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.195487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.195547 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.195842 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:42.195849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:42.196020 (+ 171us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.196159 (+ 139us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.196326 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20240503 08:13:42.197000 8375 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34345
I20240503 08:13:42.197090 8375 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:42.197331 8375 heartbeater.cc:511] Master 127.6.68.254:34345 requested a full tablet report, sending...
I20240503 08:13:42.197867 8167 ts_manager.cc:194] Registered new tserver with Master: 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791)
I20240503 08:13:42.198885 8167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:33353
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
I20240503 08:13:42.208559 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.198551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:43035 (local address 127.6.68.254:34345)
0503 08:13:42.198606 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.198607 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.198614 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:13:42.198631 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.198631 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.198639 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.198687 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:42.203793 (+ 5106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.204306 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.204941 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.205061 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.205833 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:42.205837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:42.205838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:42.205844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:42.207450 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.207873 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.207874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.207874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.207895 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.208061 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.208062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.208062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.208159 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:42.208219 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.208362 (+ 143us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.208455 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:13:42.208573 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.198124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34345 (local address 127.6.68.193:43035)
0503 08:13:42.198261 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.198273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.198332 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.198708 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.198709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.198865 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:42.199081 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.199083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.204404 (+ 5321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.204406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.204857 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.204867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.204933 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.205485 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:42.205489 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:42.205745 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:42.207544 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.207545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.207546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.207799 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.207965 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.207967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.207967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.208005 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.208218 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:42.208219 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:42.208264 (+ 45us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.208333 (+ 69us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.208454 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:13:42.223582 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34345
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:42.305982 8385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:42.306205 8385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:42.306231 8385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:42.307763 8385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:42.307817 8385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:42.307839 8385 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240503 08:13:42.307852 8385 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240503 08:13:42.307868 8385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:13:42.309520 8385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.68.254:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:42.309752 8385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:42.309924 8385 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:42.315045 8391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:42.315711 8393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:42.316587 8390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:42.317276 8385 server_base.cc:1034] running on GCE node
I20240503 08:13:42.317497 8385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:42.317739 8385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:42.318892 8385 hybrid_clock.cc:648] HybridClock initialized: now 1714724022318874 us; error 39 us; skew 500 ppm
May 03 08:13:42 dist-test-slave-3g67 krb5kdc[8126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:42.321537 8385 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:13:42.321959 8385 webserver.cc:460] Webserver started at http://127.6.68.194:40885/ using document root <none> and password file <none>
I20240503 08:13:42.322130 8385 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:42.322168 8385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:42.322257 8385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:42.323256 8385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "529d5e1c5824400bb88f611330d183bd"
format_stamp: "Formatted at 2024-05-03 08:13:42 on dist-test-slave-3g67"
server_key: "ca26a6e6586b81f6cf5246987b70cb1e"
server_key_iv: "75efaee7bec8782579c557fa3326ea51"
server_key_version: "encryptionkey@0"
I20240503 08:13:42.323565 8385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "529d5e1c5824400bb88f611330d183bd"
format_stamp: "Formatted at 2024-05-03 08:13:42 on dist-test-slave-3g67"
server_key: "ca26a6e6586b81f6cf5246987b70cb1e"
server_key_iv: "75efaee7bec8782579c557fa3326ea51"
server_key_version: "encryptionkey@0"
I20240503 08:13:42.324950 8385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:42.327546 8400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:42.327839 8385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:42.327922 8385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "529d5e1c5824400bb88f611330d183bd"
format_stamp: "Formatted at 2024-05-03 08:13:42 on dist-test-slave-3g67"
server_key: "ca26a6e6586b81f6cf5246987b70cb1e"
server_key_iv: "75efaee7bec8782579c557fa3326ea51"
server_key_version: "encryptionkey@0"
I20240503 08:13:42.328011 8385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:42.337101 8385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:42.339246 8385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:42.339372 8385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:42.339922 8385 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:42.339967 8385 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:42.340001 8385 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:42.340045 8385 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 03 08:13:42 dist-test-slave-3g67 krb5kdc[8126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:42.346485 8385 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:37789
I20240503 08:13:42.346565 8515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:37789 every 8 connection(s)
I20240503 08:13:42.346886 8385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:13:42.348146 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 8385
I20240503 08:13:42.348273 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:13:42.348453 6419 external_mini_cluster.cc:1428] Setting key e00c8ccc7241abdce5786cb2515ae134
I20240503 08:13:42.349707 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.340306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:49871 (local address 127.6.68.254:34345)
0503 08:13:42.340369 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.340372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.340556 (+ 184us) server_negotiation.cc:407] Connection header received
0503 08:13:42.341071 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.341073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.341088 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.341170 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:42.342268 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.343126 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.344000 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.344216 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.346856 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:42.346861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:42.346861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:42.346868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:42.348337 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.348768 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.348769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.348770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.348799 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.348988 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.348989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.348990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.349087 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:42.349148 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.349509 (+ 361us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.349607 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:13:42.349826 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.340283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34345 (local address 127.6.68.194:49871)
0503 08:13:42.340437 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.340465 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.340967 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.341261 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.341265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.341664 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:42.342117 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.342123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.343225 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.343226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.343893 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.343896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.344047 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.344814 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:42.344830 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:42.346723 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:42.348421 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.348427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.348430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.348667 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.348857 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.348857 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.348858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.348933 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.349144 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:42.349149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:42.349337 (+ 188us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.349469 (+ 132us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.349618 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20240503 08:13:42.350347 8516 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34345
I20240503 08:13:42.350456 8516 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:42.350714 8516 heartbeater.cc:511] Master 127.6.68.254:34345 requested a full tablet report, sending...
I20240503 08:13:42.351123 8171 ts_manager.cc:194] Registered new tserver with Master: 529d5e1c5824400bb88f611330d183bd (127.6.68.194:37789)
I20240503 08:13:42.351713 8171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:49871
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
I20240503 08:13:42.357520 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.351465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:41051 (local address 127.6.68.254:34345)
0503 08:13:42.351504 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.351506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.351643 (+ 137us) server_negotiation.cc:407] Connection header received
0503 08:13:42.351701 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.351702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.351712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.351762 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:42.352364 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.353147 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.354000 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.354114 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.355099 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:42.355102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:42.355103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:42.355108 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:42.356559 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.356923 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.356924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.356925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.356950 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.357104 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.357105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.357105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.357216 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:42.357264 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.357389 (+ 125us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.357456 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240503 08:13:42.357525 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.351416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34345 (local address 127.6.68.194:41051)
0503 08:13:42.351546 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.351556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.351639 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.351775 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.351776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.351996 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:42.352266 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.352269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.353256 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.353258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.353907 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.353910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.353991 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.354630 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:42.354634 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:42.355004 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:42.356645 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.356646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.356646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.356848 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.357010 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.357011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.357011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.357048 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.357268 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:42.357269 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:42.357306 (+ 37us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.357361 (+ 55us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.357442 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:13:42.376294 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34345
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:42.457603 8525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:42.457825 8525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:42.457852 8525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:42.459251 8525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:42.459311 8525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:42.459331 8525 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240503 08:13:42.459343 8525 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240503 08:13:42.459360 8525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:13:42.461062 8525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.68.254:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:42.461298 8525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:42.461498 8525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:42.466243 8530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:42.466885 8531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:42.467531 8533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:42.468596 8525 server_base.cc:1034] running on GCE node
I20240503 08:13:42.468807 8525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:42.469043 8525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:42.470186 8525 hybrid_clock.cc:648] HybridClock initialized: now 1714724022470178 us; error 35 us; skew 500 ppm
May 03 08:13:42 dist-test-slave-3g67 krb5kdc[8126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:42.472786 8525 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:13:42.473186 8525 webserver.cc:460] Webserver started at http://127.6.68.195:36765/ using document root <none> and password file <none>
I20240503 08:13:42.473371 8525 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:42.473414 8525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:42.473520 8525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:42.474553 8525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "5caf60ab37dd48bfae8ab00a9208a107"
format_stamp: "Formatted at 2024-05-03 08:13:42 on dist-test-slave-3g67"
server_key: "4be09ccf80ece522996256ffb282edb1"
server_key_iv: "53a7a74ffdabb8fbb054b7308e88bc25"
server_key_version: "encryptionkey@0"
I20240503 08:13:42.474857 8525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "5caf60ab37dd48bfae8ab00a9208a107"
format_stamp: "Formatted at 2024-05-03 08:13:42 on dist-test-slave-3g67"
server_key: "4be09ccf80ece522996256ffb282edb1"
server_key_iv: "53a7a74ffdabb8fbb054b7308e88bc25"
server_key_version: "encryptionkey@0"
I20240503 08:13:42.476243 8525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:13:42.476960 8540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:42.477126 8525 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:42.477208 8525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "5caf60ab37dd48bfae8ab00a9208a107"
format_stamp: "Formatted at 2024-05-03 08:13:42 on dist-test-slave-3g67"
server_key: "4be09ccf80ece522996256ffb282edb1"
server_key_iv: "53a7a74ffdabb8fbb054b7308e88bc25"
server_key_version: "encryptionkey@0"
I20240503 08:13:42.477275 8525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:42.494123 8525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:42.496330 8525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:42.496474 8525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:42.496989 8525 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:42.497035 8525 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:42.497072 8525 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:42.497103 8525 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 03 08:13:42 dist-test-slave-3g67 krb5kdc[8126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:42.503759 8525 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:34343
I20240503 08:13:42.504082 8656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:34343 every 8 connection(s)
I20240503 08:13:42.504154 8525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:13:42.506326 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.497469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:34367 (local address 127.6.68.254:34345)
0503 08:13:42.497549 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.497552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.497793 (+ 241us) server_negotiation.cc:407] Connection header received
0503 08:13:42.498568 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.498570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.498585 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.498659 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:42.499962 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.500479 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.501101 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.501270 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.503463 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:42.503469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:42.503470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:42.503478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:42.505208 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.505586 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.505587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.505587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.505604 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.505792 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.505793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.505793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.505891 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:42.505938 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.506198 (+ 260us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.506261 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240503 08:13:42.506500 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.497507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34345 (local address 127.6.68.195:34367)
0503 08:13:42.497665 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.497694 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.498414 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.498729 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.498733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.499290 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:42.499837 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.499844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.500567 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.500568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.501023 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.501025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.501167 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.501853 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:42.501875 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:42.503352 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:42.505298 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.505303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.505306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.505512 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.505655 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.505656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.505656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.505741 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.505963 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:42.505967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:42.506087 (+ 120us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.506181 (+ 94us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.506302 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20240503 08:13:42.506985 8657 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34345
I20240503 08:13:42.507062 8657 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:42.507321 8657 heartbeater.cc:511] Master 127.6.68.254:34345 requested a full tablet report, sending...
I20240503 08:13:42.507743 8171 ts_manager.cc:194] Registered new tserver with Master: 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343)
I20240503 08:13:42.508248 8171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:34367
I20240503 08:13:42.511350 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 8525
I20240503 08:13:42.511462 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:13:42.511709 6419 external_mini_cluster.cc:1428] Setting key 61cab6e5aac6cf08b3487cd598a8c79b
I20240503 08:13:42.512825 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:13:42.513767 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.508166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:59697 (local address 127.6.68.254:34345)
0503 08:13:42.508216 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.508218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.508287 (+ 69us) server_negotiation.cc:407] Connection header received
0503 08:13:42.508350 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.508351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.508361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.508422 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:42.508942 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.509446 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.510020 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.510196 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.510946 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:42.510950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:42.510950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:42.510957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:42.512848 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.513147 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.513148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.513148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.513171 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.513303 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.513305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.513305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.513400 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:42.513458 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.513590 (+ 132us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.513668 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240503 08:13:42.513767 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.508095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34345 (local address 127.6.68.195:59697)
0503 08:13:42.508228 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.508239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.508298 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.508531 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.508533 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.508673 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:42.508864 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.508865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.509518 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.509519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.509950 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.509952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.510017 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.510571 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:42.510575 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:42.510831 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:42.512935 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.512936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.512937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.513083 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.513222 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.513223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.513223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.513251 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.513455 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:42.513456 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:42.513500 (+ 44us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.513560 (+ 60us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.513668 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
May 03 08:13:42 dist-test-slave-3g67 krb5kdc[8126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 03 08:13:42 dist-test-slave-3g67 krb5kdc[8126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:42.534808 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.528287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.6.68.254:34345)
0503 08:13:42.528359 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.528361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.528371 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:13:42.528465 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.528468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.528478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.528528 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:42.529258 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.529733 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.530308 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.530491 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.532606 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:42.532612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:42.532612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:42.532619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:42.533826 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.534132 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.534133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.534134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.534157 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.534294 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.534295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.534295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.534395 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:42.534494 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.534604 (+ 110us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.534694 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240503 08:13:42.536517 8171 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:13:42.538033 8171 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:13:42.543202 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.540222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:34343 (local address 127.0.0.1:48060)
0503 08:13:42.540550 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.540576 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.540655 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.541018 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.541019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.541021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:42.541239 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.541241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.542216 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.542217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.542899 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.542900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.542974 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.543020 (+ 46us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.543077 (+ 57us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.543116 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":58,"threads_started":1}
I20240503 08:13:42.543736 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.540192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:37789 (local address 127.0.0.1:44352)
0503 08:13:42.540548 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.540577 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.540676 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.540927 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.540930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.540932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:42.541215 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.541218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.542582 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.542583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.543489 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.543491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.543553 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.543557 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.543589 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.543663 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":74,"threads_started":1}
I20240503 08:13:42.544039 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.540471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.6.68.195:34343)
0503 08:13:42.540729 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.540736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.540745 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:13:42.540786 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.540788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.540816 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.540900 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:42.541325 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.542136 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.543013 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.543838 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.543902 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.543934 (+ 32us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.543972 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":76,"threads_started":1}
I20240503 08:13:42.544477 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.540284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.6.68.194:37789)
0503 08:13:42.540464 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.540473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.540645 (+ 172us) server_negotiation.cc:407] Connection header received
0503 08:13:42.540802 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.540804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.540818 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.540890 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:42.541286 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.542486 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.543823 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.544326 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.544361 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.544394 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.544424 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20240503 08:13:42.544576 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.540536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.6.68.193:39791)
0503 08:13:42.540828 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.540834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.540845 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:13:42.540880 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.540882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.540897 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.540985 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:42.541532 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.542653 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.543856 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.544388 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.544434 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.544467 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.544505 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":91,"threads_started":1}
I20240503 08:13:42.544734 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.540427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39791 (local address 127.0.0.1:48118)
0503 08:13:42.540607 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.540615 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.540655 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.541063 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.541064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.541065 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:42.541452 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.541454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.542742 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.542743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.543598 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.543600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.543684 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.543687 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.543717 (+ 30us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.543756 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20240503 08:13:42.544679 8591 tablet_service.cc:1467] Processing CreateTablet for tablet 77a3ce2c183143fdb0bea2ba0d5ef17e (DEFAULT_TABLE table=test-table [id=ccb3917d1c8e4b9a8e259f397d3a1ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:42.545015 8591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a3ce2c183143fdb0bea2ba0d5ef17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:42.544998 8450 tablet_service.cc:1467] Processing CreateTablet for tablet 77a3ce2c183143fdb0bea2ba0d5ef17e (DEFAULT_TABLE table=test-table [id=ccb3917d1c8e4b9a8e259f397d3a1ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:42.545284 8450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a3ce2c183143fdb0bea2ba0d5ef17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:42.545285 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 77a3ce2c183143fdb0bea2ba0d5ef17e (DEFAULT_TABLE table=test-table [id=ccb3917d1c8e4b9a8e259f397d3a1ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:42.545564 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a3ce2c183143fdb0bea2ba0d5ef17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:42.547521 8680 tablet_bootstrap.cc:492] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: Bootstrap starting.
I20240503 08:13:42.548388 8681 tablet_bootstrap.cc:492] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: Bootstrap starting.
I20240503 08:13:42.548682 8680 tablet_bootstrap.cc:654] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:42.549029 8680 log.cc:824] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:42.549422 8682 tablet_bootstrap.cc:492] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: Bootstrap starting.
I20240503 08:13:42.549494 8681 tablet_bootstrap.cc:654] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:42.549777 8681 log.cc:824] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:42.549850 8680 tablet_bootstrap.cc:492] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: No bootstrap required, opened a new log
I20240503 08:13:42.549925 8680 ts_tablet_manager.cc:1367] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:42.550530 8681 tablet_bootstrap.cc:492] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: No bootstrap required, opened a new log
I20240503 08:13:42.550597 8681 ts_tablet_manager.cc:1367] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:42.551807 8680 raft_consensus.cc:348] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.551970 8680 raft_consensus.cc:374] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:42.551995 8680 raft_consensus.cc:729] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5caf60ab37dd48bfae8ab00a9208a107, State: Initialized, Role: FOLLOWER
I20240503 08:13:42.552114 8680 consensus_queue.cc:260] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.552381 8680 ts_tablet_manager.cc:1398] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:42.552431 8657 heartbeater.cc:503] Master 127.6.68.254:34345 was elected leader, sending a full tablet report...
I20240503 08:13:42.552469 8681 raft_consensus.cc:348] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.552573 8681 raft_consensus.cc:374] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:42.552589 8681 raft_consensus.cc:729] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529d5e1c5824400bb88f611330d183bd, State: Initialized, Role: FOLLOWER
I20240503 08:13:42.552659 8681 consensus_queue.cc:260] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.552872 8682 tablet_bootstrap.cc:654] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:42.552907 8681 ts_tablet_manager.cc:1398] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:42.553035 8516 heartbeater.cc:503] Master 127.6.68.254:34345 was elected leader, sending a full tablet report...
I20240503 08:13:42.553213 8682 log.cc:824] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:42.554054 8682 tablet_bootstrap.cc:492] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: No bootstrap required, opened a new log
I20240503 08:13:42.554118 8682 ts_tablet_manager.cc:1367] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:13:42.555546 8682 raft_consensus.cc:348] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.555706 8682 raft_consensus.cc:374] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:42.555733 8682 raft_consensus.cc:729] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b6072e21c84f568a1d5fc931e1884f, State: Initialized, Role: FOLLOWER
I20240503 08:13:42.555835 8682 consensus_queue.cc:260] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.556047 8682 ts_tablet_manager.cc:1398] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:42.556113 8375 heartbeater.cc:503] Master 127.6.68.254:34345 was elected leader, sending a full tablet report...
W20240503 08:13:42.597465 8517 tablet.cc:2369] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:13:42.695207 8376 tablet.cc:2369] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:13:42.732720 8688 raft_consensus.cc:482] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:42.732856 8688 raft_consensus.cc:504] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.733235 8688 leader_election.cc:288] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 529d5e1c5824400bb88f611330d183bd (127.6.68.194:37789), 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343)
I20240503 08:13:42.736362 8689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.733439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:37789 (local address 127.6.68.193:34973)
0503 08:13:42.733597 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.733607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.733685 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.733855 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.733855 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.733856 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:42.734116 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.734118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.735088 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.735089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.736070 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.736074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.736157 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.736163 (+ 6us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.736208 (+ 45us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.736242 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20240503 08:13:42.736362 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.733454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:34343 (local address 127.6.68.193:33571)
0503 08:13:42.733615 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.733626 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.733706 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.733852 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.733854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.733856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:42.734116 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.734118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.735088 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.735089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.736070 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.736074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.736158 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.736163 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.736199 (+ 36us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.736242 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":60,"threads_started":1}
I20240503 08:13:42.736860 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.733555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:33571 (local address 127.6.68.195:34343)
0503 08:13:42.733599 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.733601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.733688 (+ 87us) server_negotiation.cc:407] Connection header received
0503 08:13:42.733762 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.733763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.733772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.733835 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:42.734197 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.734986 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.736212 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.736725 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.736734 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.736760 (+ 26us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.736792 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:13:42.736860 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.733511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:34973 (local address 127.6.68.194:37789)
0503 08:13:42.733561 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.733564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.733683 (+ 119us) server_negotiation.cc:407] Connection header received
0503 08:13:42.733751 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.733752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.733763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.733835 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:42.734189 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.734980 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.736272 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.736738 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.736746 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.736765 (+ 19us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.736793 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240503 08:13:42.737243 8470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3ce2c183143fdb0bea2ba0d5ef17e" candidate_uuid: "17b6072e21c84f568a1d5fc931e1884f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529d5e1c5824400bb88f611330d183bd" is_pre_election: true
I20240503 08:13:42.737282 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3ce2c183143fdb0bea2ba0d5ef17e" candidate_uuid: "17b6072e21c84f568a1d5fc931e1884f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5caf60ab37dd48bfae8ab00a9208a107" is_pre_election: true
I20240503 08:13:42.737422 8470 raft_consensus.cc:2454] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b6072e21c84f568a1d5fc931e1884f in term 0.
I20240503 08:13:42.737422 8611 raft_consensus.cc:2454] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b6072e21c84f568a1d5fc931e1884f in term 0.
I20240503 08:13:42.737663 8260 leader_election.cc:302] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b6072e21c84f568a1d5fc931e1884f, 5caf60ab37dd48bfae8ab00a9208a107; no voters:
I20240503 08:13:42.737834 8688 raft_consensus.cc:2781] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:42.737915 8688 raft_consensus.cc:482] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:42.737937 8688 raft_consensus.cc:3037] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:42.738648 8688 raft_consensus.cc:504] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.738813 8688 leader_election.cc:288] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [CANDIDATE]: Term 1 election: Requested vote from peers 529d5e1c5824400bb88f611330d183bd (127.6.68.194:37789), 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343)
I20240503 08:13:42.739017 8470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3ce2c183143fdb0bea2ba0d5ef17e" candidate_uuid: "17b6072e21c84f568a1d5fc931e1884f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529d5e1c5824400bb88f611330d183bd"
I20240503 08:13:42.739046 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3ce2c183143fdb0bea2ba0d5ef17e" candidate_uuid: "17b6072e21c84f568a1d5fc931e1884f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5caf60ab37dd48bfae8ab00a9208a107"
I20240503 08:13:42.739089 8470 raft_consensus.cc:3037] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:42.739118 8611 raft_consensus.cc:3037] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:42.739868 8611 raft_consensus.cc:2454] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b6072e21c84f568a1d5fc931e1884f in term 1.
I20240503 08:13:42.739872 8470 raft_consensus.cc:2454] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b6072e21c84f568a1d5fc931e1884f in term 1.
I20240503 08:13:42.740036 8260 leader_election.cc:302] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b6072e21c84f568a1d5fc931e1884f, 5caf60ab37dd48bfae8ab00a9208a107; no voters:
I20240503 08:13:42.740128 8688 raft_consensus.cc:2781] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:42.740236 8688 raft_consensus.cc:686] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 1 LEADER]: Becoming Leader. State: Replica: 17b6072e21c84f568a1d5fc931e1884f, State: Running, Role: LEADER
I20240503 08:13:42.740330 8688 consensus_queue.cc:237] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.741133 8171 catalog_manager.cc:5484] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193). New cstate: current_term: 1 leader_uuid: "17b6072e21c84f568a1d5fc931e1884f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } health_report { overall_health: UNKNOWN } } }
W20240503 08:13:42.754758 8658 tablet.cc:2369] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: Can't schedule compaction. Clean time has not been advanced past its initial value.
May 03 08:13:42 dist-test-slave-3g67 krb5kdc[8126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724021, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:42.799247 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.792530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34345 (local address 127.0.0.1:36514)
0503 08:13:42.792744 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.792755 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.792836 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.792983 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.792984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.793401 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:42.793561 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.793565 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.794224 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.794225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.794575 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.794576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.794641 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.795256 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:42.795260 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:42.796725 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:42.798386 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.798392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.798396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.798591 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.798770 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:42.798771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:42.798771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:42.798794 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:42.798974 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:42.798977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:42.799019 (+ 42us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.799067 (+ 48us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.799162 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20240503 08:13:42.799327 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.792624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.6.68.254:34345)
0503 08:13:42.792740 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.792742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.792828 (+ 86us) server_negotiation.cc:407] Connection header received
0503 08:13:42.792901 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.792902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.792912 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.792971 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:42.793663 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.794140 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.794643 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.794805 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.796844 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:42.796850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:42.796850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:42.796856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:42.798302 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.798676 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.798677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.798677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.798711 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:42.798839 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:42.798840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:42.798840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:42.798931 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:42.798979 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.799093 (+ 114us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.799223 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":63}
I20240503 08:13:42.801286 8171 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:36514:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240503 08:13:42.801468 8171 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:13:42.802981 8591 tablet_service.cc:1467] Processing CreateTablet for tablet 4f517b4deae1444e9677b348d57bd8ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4289c572521e46c691b6dc507dc2a867]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240503 08:13:42.802991 8450 tablet_service.cc:1467] Processing CreateTablet for tablet 4f517b4deae1444e9677b348d57bd8ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4289c572521e46c691b6dc507dc2a867]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240503 08:13:42.802991 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 4f517b4deae1444e9677b348d57bd8ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4289c572521e46c691b6dc507dc2a867]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240503 08:13:42.803140 8591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f517b4deae1444e9677b348d57bd8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:42.803143 8450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f517b4deae1444e9677b348d57bd8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:42.803141 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f517b4deae1444e9677b348d57bd8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:42.804268 8680 tablet_bootstrap.cc:492] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107: Bootstrap starting.
I20240503 08:13:42.804630 8680 tablet_bootstrap.cc:654] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:42.804641 8681 tablet_bootstrap.cc:492] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: Bootstrap starting.
I20240503 08:13:42.804672 8682 tablet_bootstrap.cc:492] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f: Bootstrap starting.
I20240503 08:13:42.805020 8682 tablet_bootstrap.cc:654] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:42.805066 8681 tablet_bootstrap.cc:654] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:42.805696 8680 tablet_bootstrap.cc:492] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107: No bootstrap required, opened a new log
I20240503 08:13:42.805720 8682 tablet_bootstrap.cc:492] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f: No bootstrap required, opened a new log
I20240503 08:13:42.805742 8680 ts_tablet_manager.cc:1367] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:42.805778 8682 ts_tablet_manager.cc:1367] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:42.805936 8682 raft_consensus.cc:348] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.805927 8680 raft_consensus.cc:348] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.805995 8682 raft_consensus.cc:374] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:42.806002 8680 raft_consensus.cc:374] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:42.806010 8682 raft_consensus.cc:729] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b6072e21c84f568a1d5fc931e1884f, State: Initialized, Role: FOLLOWER
I20240503 08:13:42.806023 8680 raft_consensus.cc:729] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5caf60ab37dd48bfae8ab00a9208a107, State: Initialized, Role: FOLLOWER
I20240503 08:13:42.806020 8681 tablet_bootstrap.cc:492] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: No bootstrap required, opened a new log
I20240503 08:13:42.806061 8681 ts_tablet_manager.cc:1367] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:42.806049 8682 consensus_queue.cc:260] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.806067 8680 consensus_queue.cc:260] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.806141 8682 ts_tablet_manager.cc:1398] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:42.806159 8680 ts_tablet_manager.cc:1398] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:42.806167 8688 tablet_replica.cc:439] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } } }
I20240503 08:13:42.806237 8688 tablet_replica.cc:442] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:42.806211 8681 raft_consensus.cc:348] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.806272 8681 raft_consensus.cc:374] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:42.806293 8681 raft_consensus.cc:729] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529d5e1c5824400bb88f611330d183bd, State: Initialized, Role: FOLLOWER
I20240503 08:13:42.806268 8685 tablet_replica.cc:439] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } } }
I20240503 08:13:42.806347 8685 tablet_replica.cc:442] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:42.806339 8681 consensus_queue.cc:260] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.806438 8681 ts_tablet_manager.cc:1398] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:42.806479 8686 tablet_replica.cc:439] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } } }
I20240503 08:13:42.806568 8686 tablet_replica.cc:442] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:42.881728 8686 raft_consensus.cc:482] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:42.881842 8686 raft_consensus.cc:504] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.882181 8686 leader_election.cc:288] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791), 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343)
I20240503 08:13:42.884474 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.882355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39791 (local address 127.6.68.194:51417)
0503 08:13:42.882519 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.882529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.882599 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.882747 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.882748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.882749 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:42.882916 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.882917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.883546 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.883547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.884228 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.884230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.884303 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.884309 (+ 6us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.884341 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.884378 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20240503 08:13:42.884609 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.882355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:34343 (local address 127.6.68.194:55769)
0503 08:13:42.882505 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.882514 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.882573 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.882709 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.882710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.882712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:42.882954 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.882956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.883675 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.883676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.884383 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.884384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.884455 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.884458 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.884489 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.884524 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20240503 08:13:42.884920 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.882423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:51417 (local address 127.6.68.193:39791)
0503 08:13:42.882469 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.882471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.882605 (+ 134us) server_negotiation.cc:407] Connection header received
0503 08:13:42.882661 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.882662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.882670 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.882738 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:42.882971 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.883481 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.884318 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.884822 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.884830 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.884849 (+ 19us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.884873 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:13:42.884974 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.882423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:55769 (local address 127.6.68.195:34343)
0503 08:13:42.882469 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.882472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.882579 (+ 107us) server_negotiation.cc:407] Connection header received
0503 08:13:42.882630 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.882631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.882641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.882693 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:42.883028 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.883570 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.884464 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.884878 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.884885 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.884906 (+ 21us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.884928 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:13:42.885241 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f517b4deae1444e9677b348d57bd8ff" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b6072e21c84f568a1d5fc931e1884f" is_pre_election: true
I20240503 08:13:42.885370 8329 raft_consensus.cc:2454] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 0.
I20240503 08:13:42.885396 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f517b4deae1444e9677b348d57bd8ff" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5caf60ab37dd48bfae8ab00a9208a107" is_pre_election: true
I20240503 08:13:42.885500 8611 raft_consensus.cc:2454] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 0.
I20240503 08:13:42.885574 8403 leader_election.cc:302] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b6072e21c84f568a1d5fc931e1884f, 529d5e1c5824400bb88f611330d183bd; no voters:
I20240503 08:13:42.885713 8686 raft_consensus.cc:2781] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:42.885784 8686 raft_consensus.cc:482] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:42.885802 8686 raft_consensus.cc:3037] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:42.886494 8686 raft_consensus.cc:504] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.886636 8686 leader_election.cc:288] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 election: Requested vote from peers 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791), 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343)
I20240503 08:13:42.886891 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f517b4deae1444e9677b348d57bd8ff" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5caf60ab37dd48bfae8ab00a9208a107"
I20240503 08:13:42.886983 8611 raft_consensus.cc:3037] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:42.887137 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f517b4deae1444e9677b348d57bd8ff" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b6072e21c84f568a1d5fc931e1884f"
I20240503 08:13:42.887212 8329 raft_consensus.cc:3037] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:42.887936 8611 raft_consensus.cc:2454] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 1.
I20240503 08:13:42.888125 8401 leader_election.cc:302] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529d5e1c5824400bb88f611330d183bd, 5caf60ab37dd48bfae8ab00a9208a107; no voters:
I20240503 08:13:42.888211 8329 raft_consensus.cc:2454] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 1.
I20240503 08:13:42.888237 8686 raft_consensus.cc:2781] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:42.888414 8686 raft_consensus.cc:686] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [term 1 LEADER]: Becoming Leader. State: Replica: 529d5e1c5824400bb88f611330d183bd, State: Running, Role: LEADER
I20240503 08:13:42.888515 8686 consensus_queue.cc:237] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.888986 8704 tablet_replica.cc:439] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: TxnStatusTablet state changed. Reason: New leader 529d5e1c5824400bb88f611330d183bd. Latest consensus state: current_term: 1 leader_uuid: "529d5e1c5824400bb88f611330d183bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } } }
I20240503 08:13:42.889091 8704 tablet_replica.cc:442] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: This TxnStatusTablet replica's current role is: LEADER
I20240503 08:13:42.889288 8706 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240503 08:13:42.889392 8706 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240503 08:13:42.889431 8170 catalog_manager.cc:5484] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 529d5e1c5824400bb88f611330d183bd (127.6.68.194). New cstate: current_term: 1 leader_uuid: "529d5e1c5824400bb88f611330d183bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.903483 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.900883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:37789 (local address 127.0.0.1:44362)
0503 08:13:42.901000 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.901014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.901072 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.901285 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.901286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.901288 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0503 08:13:42.901450 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.901452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.902381 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.902382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.902813 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.902814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.902874 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.902892 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0503 08:13:42.903296 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0503 08:13:42.903307 (+ 11us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.903358 (+ 51us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.903404 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20240503 08:13:42.903483 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.900988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.6.68.194:37789)
0503 08:13:42.901050 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.901053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.901064 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:13:42.901153 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.901155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.901174 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.901253 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:42.901516 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.902291 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.902887 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.903063 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.903098 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:42.903206 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:42.903286 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.903375 (+ 89us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.903402 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240503 08:13:42.905277 8611 raft_consensus.cc:1261] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Refusing update from remote peer 529d5e1c5824400bb88f611330d183bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:42.905277 8329 raft_consensus.cc:1261] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Refusing update from remote peer 529d5e1c5824400bb88f611330d183bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:42.905602 8686 consensus_queue.cc:1035] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:42.905727 8704 consensus_queue.cc:1035] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:42.907002 8685 tablet_replica.cc:439] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107: TxnStatusTablet state changed. Reason: New leader 529d5e1c5824400bb88f611330d183bd. Latest consensus state: current_term: 1 leader_uuid: "529d5e1c5824400bb88f611330d183bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } } }
I20240503 08:13:42.907104 8685 tablet_replica.cc:442] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:42.907400 8688 tablet_replica.cc:439] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f: TxnStatusTablet state changed. Reason: New leader 529d5e1c5824400bb88f611330d183bd. Latest consensus state: current_term: 1 leader_uuid: "529d5e1c5824400bb88f611330d183bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } } }
I20240503 08:13:42.907495 8688 tablet_replica.cc:442] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:42.907967 8686 tablet_replica.cc:439] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "529d5e1c5824400bb88f611330d183bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } } }
I20240503 08:13:42.908051 8686 tablet_replica.cc:442] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: This TxnStatusTablet replica's current role is: LEADER
I20240503 08:13:42.908584 8685 tablet_replica.cc:439] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "529d5e1c5824400bb88f611330d183bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } } }
I20240503 08:13:42.908679 8685 tablet_replica.cc:442] T 4f517b4deae1444e9677b348d57bd8ff P 5caf60ab37dd48bfae8ab00a9208a107: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:42.909822 8686 tablet_replica.cc:439] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "529d5e1c5824400bb88f611330d183bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } } }
I20240503 08:13:42.909919 8686 tablet_replica.cc:442] T 4f517b4deae1444e9677b348d57bd8ff P 529d5e1c5824400bb88f611330d183bd: This TxnStatusTablet replica's current role is: LEADER
I20240503 08:13:42.910426 8688 tablet_replica.cc:439] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "529d5e1c5824400bb88f611330d183bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } } }
I20240503 08:13:42.910511 8688 tablet_replica.cc:442] T 4f517b4deae1444e9677b348d57bd8ff P 17b6072e21c84f568a1d5fc931e1884f: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:13:42.912905 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.910269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.6.68.193:39791)
0503 08:13:42.910318 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.910320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.910333 (+ 13us) server_negotiation.cc:407] Connection header received
0503 08:13:42.910720 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.910721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.910741 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.910818 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:42.911141 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.911606 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.912455 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.912582 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.912617 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:42.912701 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:42.912764 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.912826 (+ 62us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.912848 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:13:42.917590 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.915155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:37789 (local address 127.6.68.193:38615)
0503 08:13:42.915309 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.915320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.915385 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.915507 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.915508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.915509 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0503 08:13:42.915723 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.915724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.916627 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.916628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.917072 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.917074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.917155 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.917173 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0503 08:13:42.917442 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0503 08:13:42.917443 (+ 1us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.917479 (+ 36us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.917511 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20240503 08:13:42.917584 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.915243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:38615 (local address 127.6.68.194:37789)
0503 08:13:42.915287 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.915289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.915386 (+ 97us) server_negotiation.cc:407] Connection header received
0503 08:13:42.915427 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.915428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.915438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.915502 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:42.915785 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.916505 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.917148 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.917315 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.917334 (+ 19us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:42.917379 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:42.917437 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.917496 (+ 59us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.917517 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240503 08:13:42.920008 8711 mvcc.cc:205] Tried to move back new op lower bound from 7023509597817556992 to 7023509597751963648. Current Snapshot: MvccSnapshot[applied={T|T < 7023509597817556992}]
I20240503 08:13:42.920401 8611 raft_consensus.cc:1261] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Refusing update from remote peer 17b6072e21c84f568a1d5fc931e1884f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:42.920502 8470 raft_consensus.cc:1261] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Refusing update from remote peer 17b6072e21c84f568a1d5fc931e1884f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:42.920773 8691 consensus_queue.cc:1035] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:42.920943 8688 consensus_queue.cc:1035] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [LEADER]: Connected to new peer: Peer: permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:42.922362 8710 mvcc.cc:205] Tried to move back new op lower bound from 7023509597880127488 to 7023509597144993792. Current Snapshot: MvccSnapshot[applied={T|T < 7023509597880127488}]
I20240503 08:13:42.923175 8709 mvcc.cc:205] Tried to move back new op lower bound from 7023509597880127488 to 7023509597144993792. Current Snapshot: MvccSnapshot[applied={T|T < 7023509597880127488}]
I20240503 08:13:42.930356 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.927673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:60165 (local address 127.6.68.193:39791)
0503 08:13:42.927727 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:42.927729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:42.927821 (+ 92us) server_negotiation.cc:407] Connection header received
0503 08:13:42.927921 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:42.927922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:42.927949 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:42.928005 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:42.928354 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.928811 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.929662 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.929862 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.930085 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:42.930121 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:42.930170 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:42.930245 (+ 75us) server_negotiation.cc:299] Negotiation successful
0503 08:13:42.930274 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240503 08:13:42.930393 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:42.927545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39791 (local address 127.6.68.194:60165)
0503 08:13:42.927742 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:42.927754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:42.927851 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:42.928019 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:42.928020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:42.928022 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0503 08:13:42.928283 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.928285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.928899 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:42.928900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:42.929558 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:42.929560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:42.929891 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:42.929913 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0503 08:13:42.930191 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0503 08:13:42.930192 (+ 1us) client_negotiation.cc:769] Sending connection context
0503 08:13:42.930241 (+ 49us) client_negotiation.cc:240] Negotiation successful
0503 08:13:42.930289 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":122,"threads_started":1}
I20240503 08:13:42.941591 8167 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36500:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:42.941738 8167 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36500:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:42.942564 8167 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079: Sending DeleteTablet for 3 replicas of tablet 77a3ce2c183143fdb0bea2ba0d5ef17e
I20240503 08:13:42.942916 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a3ce2c183143fdb0bea2ba0d5ef17e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:42 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:48118
I20240503 08:13:42.942909 8450 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a3ce2c183143fdb0bea2ba0d5ef17e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:42 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:44352
I20240503 08:13:42.943284 8731 tablet_replica.cc:330] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: stopping tablet replica
I20240503 08:13:42.943413 8731 raft_consensus.cc:2229] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:42.943512 8731 raft_consensus.cc:2258] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:42.943477 8167 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:13:42.943661 8167 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:13:42.944180 8730 tablet_replica.cc:330] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: stopping tablet replica
I20240503 08:13:42.944156 8591 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a3ce2c183143fdb0bea2ba0d5ef17e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:42 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:48060
I20240503 08:13:42.944272 8730 raft_consensus.cc:2229] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:42.944381 8730 raft_consensus.cc:2258] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:42.945125 8730 ts_tablet_manager.cc:1875] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:42.945207 8309 tablet_service.cc:1467] Processing CreateTablet for tablet c858a001fa994b73b9bb8ed99e51724a (DEFAULT_TABLE table=test-table [id=172a522e7fe44ad1897feaa46215e3b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:42.945345 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c858a001fa994b73b9bb8ed99e51724a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:42.946681 8450 tablet_service.cc:1467] Processing CreateTablet for tablet c858a001fa994b73b9bb8ed99e51724a (DEFAULT_TABLE table=test-table [id=172a522e7fe44ad1897feaa46215e3b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:42.946789 8450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c858a001fa994b73b9bb8ed99e51724a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:42.947016 8730 ts_tablet_manager.cc:1888] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:13:42.947079 8730 log.cc:1196] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/77a3ce2c183143fdb0bea2ba0d5ef17e
I20240503 08:13:42.947345 8730 ts_tablet_manager.cc:1909] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 17b6072e21c84f568a1d5fc931e1884f: Deleting consensus metadata
I20240503 08:13:42.948100 8682 tablet_bootstrap.cc:492] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f: Bootstrap starting.
I20240503 08:13:42.948496 8682 tablet_bootstrap.cc:654] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:42.948838 8681 tablet_bootstrap.cc:492] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd: Bootstrap starting.
I20240503 08:13:42.949172 8682 tablet_bootstrap.cc:492] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f: No bootstrap required, opened a new log
I20240503 08:13:42.949190 8681 tablet_bootstrap.cc:654] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:42.949218 8682 ts_tablet_manager.cc:1367] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:42.949375 8682 raft_consensus.cc:348] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.949453 8682 raft_consensus.cc:374] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:42.949472 8682 raft_consensus.cc:729] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b6072e21c84f568a1d5fc931e1884f, State: Initialized, Role: FOLLOWER
I20240503 08:13:42.949515 8682 consensus_queue.cc:260] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.949599 8682 ts_tablet_manager.cc:1398] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:42.949697 8157 catalog_manager.cc:4830] TS 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791): tablet 77a3ce2c183143fdb0bea2ba0d5ef17e (table test-table [id=ccb3917d1c8e4b9a8e259f397d3a1ecf]) successfully deleted
I20240503 08:13:42.950477 8591 tablet_service.cc:1467] Processing CreateTablet for tablet c858a001fa994b73b9bb8ed99e51724a (DEFAULT_TABLE table=test-table [id=172a522e7fe44ad1897feaa46215e3b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:42.950623 8591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c858a001fa994b73b9bb8ed99e51724a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:42.950994 8731 ts_tablet_manager.cc:1875] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:42.952054 8681 tablet_bootstrap.cc:492] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd: No bootstrap required, opened a new log
I20240503 08:13:42.952102 8681 ts_tablet_manager.cc:1367] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:13:42.952203 8680 tablet_bootstrap.cc:492] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107: Bootstrap starting.
I20240503 08:13:42.952247 8681 raft_consensus.cc:348] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.952308 8681 raft_consensus.cc:374] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:42.952328 8681 raft_consensus.cc:729] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529d5e1c5824400bb88f611330d183bd, State: Initialized, Role: FOLLOWER
I20240503 08:13:42.952369 8681 consensus_queue.cc:260] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.952446 8681 ts_tablet_manager.cc:1398] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:42.952593 8680 tablet_bootstrap.cc:654] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:42.950635 8732 tablet_replica.cc:330] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: stopping tablet replica
I20240503 08:13:42.953189 8732 raft_consensus.cc:2229] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:42.953258 8732 raft_consensus.cc:2258] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:42.953727 8732 ts_tablet_manager.cc:1875] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:42.955428 8732 ts_tablet_manager.cc:1888] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:13:42.955509 8732 log.cc:1196] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/77a3ce2c183143fdb0bea2ba0d5ef17e
I20240503 08:13:42.955796 8732 ts_tablet_manager.cc:1909] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 5caf60ab37dd48bfae8ab00a9208a107: Deleting consensus metadata
I20240503 08:13:42.956043 8731 ts_tablet_manager.cc:1888] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:13:42.956152 8731 log.cc:1196] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/77a3ce2c183143fdb0bea2ba0d5ef17e
I20240503 08:13:42.956286 8155 catalog_manager.cc:4830] TS 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343): tablet 77a3ce2c183143fdb0bea2ba0d5ef17e (table test-table [id=ccb3917d1c8e4b9a8e259f397d3a1ecf]) successfully deleted
I20240503 08:13:42.956532 8731 ts_tablet_manager.cc:1909] T 77a3ce2c183143fdb0bea2ba0d5ef17e P 529d5e1c5824400bb88f611330d183bd: Deleting consensus metadata
I20240503 08:13:42.956948 8157 catalog_manager.cc:4830] TS 529d5e1c5824400bb88f611330d183bd (127.6.68.194:37789): tablet 77a3ce2c183143fdb0bea2ba0d5ef17e (table test-table [id=ccb3917d1c8e4b9a8e259f397d3a1ecf]) successfully deleted
I20240503 08:13:42.957589 8680 tablet_bootstrap.cc:492] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107: No bootstrap required, opened a new log
I20240503 08:13:42.957649 8680 ts_tablet_manager.cc:1367] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:13:42.957818 8680 raft_consensus.cc:348] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.957896 8680 raft_consensus.cc:374] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:42.957935 8680 raft_consensus.cc:729] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5caf60ab37dd48bfae8ab00a9208a107, State: Initialized, Role: FOLLOWER
I20240503 08:13:42.957978 8680 consensus_queue.cc:260] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:42.958077 8680 ts_tablet_manager.cc:1398] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:43.012748 8685 raft_consensus.cc:482] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:43.012885 8685 raft_consensus.cc:504] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:43.013224 8685 leader_election.cc:288] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791), 529d5e1c5824400bb88f611330d183bd (127.6.68.194:37789)
I20240503 08:13:43.017055 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:43.013497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:38797 (local address 127.6.68.193:39791)
0503 08:13:43.013546 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:43.013548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:43.013832 (+ 284us) server_negotiation.cc:407] Connection header received
0503 08:13:43.013947 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:43.013949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:43.013959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:43.013988 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:43.014434 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.015146 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:43.016332 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.016782 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:43.016793 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:43.016934 (+ 141us) server_negotiation.cc:299] Negotiation successful
0503 08:13:43.016975 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:13:43.017079 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:43.013641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:54361 (local address 127.6.68.194:37789)
0503 08:13:43.013681 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:43.013684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:43.013802 (+ 118us) server_negotiation.cc:407] Connection header received
0503 08:13:43.013926 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:43.013928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:43.013938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:43.013970 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:43.014485 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.015209 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:43.016376 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.016846 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:43.016854 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:43.016973 (+ 119us) server_negotiation.cc:299] Negotiation successful
0503 08:13:43.017006 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:13:43.017336 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:43.013425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39791 (local address 127.6.68.195:38797)
0503 08:13:43.013734 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:43.013746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:43.013864 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:43.014075 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:43.014076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:43.014084 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:43.014334 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:43.014336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.015259 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:43.015259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:43.016218 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:43.016221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.016839 (+ 618us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:43.016843 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:13:43.017205 (+ 362us) client_negotiation.cc:240] Negotiation successful
0503 08:13:43.017255 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":62,"threads_started":1}
I20240503 08:13:43.017570 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c858a001fa994b73b9bb8ed99e51724a" candidate_uuid: "5caf60ab37dd48bfae8ab00a9208a107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b6072e21c84f568a1d5fc931e1884f" is_pre_election: true
I20240503 08:13:43.017678 8329 raft_consensus.cc:2454] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5caf60ab37dd48bfae8ab00a9208a107 in term 0.
I20240503 08:13:43.017791 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:43.013585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:37789 (local address 127.6.68.195:54361)
0503 08:13:43.013736 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:43.013746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:43.013831 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:43.014166 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:43.014167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:43.014169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:43.014397 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:43.014399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.015306 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:43.015307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:43.016271 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:43.016273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.016883 (+ 610us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:43.016886 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:13:43.017563 (+ 677us) client_negotiation.cc:240] Negotiation successful
0503 08:13:43.017610 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":30,"threads_started":1}
I20240503 08:13:43.017876 8543 leader_election.cc:302] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b6072e21c84f568a1d5fc931e1884f, 5caf60ab37dd48bfae8ab00a9208a107; no voters:
I20240503 08:13:43.018065 8685 raft_consensus.cc:2781] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:43.018124 8685 raft_consensus.cc:482] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:43.018146 8685 raft_consensus.cc:3037] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:43.018237 8470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c858a001fa994b73b9bb8ed99e51724a" candidate_uuid: "5caf60ab37dd48bfae8ab00a9208a107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529d5e1c5824400bb88f611330d183bd" is_pre_election: true
I20240503 08:13:43.018334 8470 raft_consensus.cc:2454] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5caf60ab37dd48bfae8ab00a9208a107 in term 0.
I20240503 08:13:43.019026 8685 raft_consensus.cc:504] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:43.019172 8685 leader_election.cc:288] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [CANDIDATE]: Term 1 election: Requested vote from peers 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791), 529d5e1c5824400bb88f611330d183bd (127.6.68.194:37789)
I20240503 08:13:43.019434 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c858a001fa994b73b9bb8ed99e51724a" candidate_uuid: "5caf60ab37dd48bfae8ab00a9208a107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b6072e21c84f568a1d5fc931e1884f"
I20240503 08:13:43.019517 8329 raft_consensus.cc:3037] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:43.019740 8470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c858a001fa994b73b9bb8ed99e51724a" candidate_uuid: "5caf60ab37dd48bfae8ab00a9208a107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529d5e1c5824400bb88f611330d183bd"
I20240503 08:13:43.019814 8470 raft_consensus.cc:3037] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:43.020380 8329 raft_consensus.cc:2454] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5caf60ab37dd48bfae8ab00a9208a107 in term 1.
I20240503 08:13:43.020556 8543 leader_election.cc:302] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b6072e21c84f568a1d5fc931e1884f, 5caf60ab37dd48bfae8ab00a9208a107; no voters:
I20240503 08:13:43.020591 8470 raft_consensus.cc:2454] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5caf60ab37dd48bfae8ab00a9208a107 in term 1.
I20240503 08:13:43.020642 8685 raft_consensus.cc:2781] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:43.020742 8685 raft_consensus.cc:686] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 LEADER]: Becoming Leader. State: Replica: 5caf60ab37dd48bfae8ab00a9208a107, State: Running, Role: LEADER
I20240503 08:13:43.020820 8685 consensus_queue.cc:237] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:43.021587 8167 catalog_manager.cc:5484] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195). New cstate: current_term: 1 leader_uuid: "5caf60ab37dd48bfae8ab00a9208a107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:43.032300 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:43.029517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.6.68.195:34343)
0503 08:13:43.029635 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:43.029638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:43.029645 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:13:43.029694 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:43.029695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:43.029714 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:43.029782 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:43.030110 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.030831 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:43.031709 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.031877 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:43.031923 (+ 46us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:43.032032 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:43.032112 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:43.032193 (+ 81us) server_negotiation.cc:299] Negotiation successful
0503 08:13:43.032217 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20240503 08:13:43.038241 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:43.034916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:37789 (local address 127.6.68.195:47019)
0503 08:13:43.035083 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:43.035095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:43.035152 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:43.035289 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:43.035290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:43.035292 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0503 08:13:43.036025 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:43.036028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.036972 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:43.036973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:43.037620 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:43.037623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.037710 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:43.037730 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0503 08:13:43.038040 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0503 08:13:43.038042 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:13:43.038097 (+ 55us) client_negotiation.cc:240] Negotiation successful
0503 08:13:43.038149 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20240503 08:13:43.038231 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:43.035101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:47019 (local address 127.6.68.194:37789)
0503 08:13:43.035141 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:43.035143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:43.035153 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:13:43.035210 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:43.035211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:43.035222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:43.035280 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:43.036111 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.036877 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:43.037734 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.037909 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:43.037932 (+ 23us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:43.037960 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:43.038021 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:43.038108 (+ 87us) server_negotiation.cc:299] Negotiation successful
0503 08:13:43.038131 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240503 08:13:43.041150 8470 raft_consensus.cc:1261] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Refusing update from remote peer 5caf60ab37dd48bfae8ab00a9208a107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:43.041155 8329 raft_consensus.cc:1261] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Refusing update from remote peer 5caf60ab37dd48bfae8ab00a9208a107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:43.041380 8685 consensus_queue.cc:1035] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:43.041555 8685 consensus_queue.cc:1035] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:43.051827 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:43.047817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:34343 (local address 127.6.68.194:52527)
0503 08:13:43.047911 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:43.047923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:43.047965 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:43.048223 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:43.048224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:43.048225 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0503 08:13:43.048438 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:43.048440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.050499 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:43.050501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:43.051139 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:43.051142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.051213 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:43.051220 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0503 08:13:43.051640 (+ 420us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0503 08:13:43.051643 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:13:43.051684 (+ 41us) client_negotiation.cc:240] Negotiation successful
0503 08:13:43.051736 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20240503 08:13:43.052596 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:43.048042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:52527 (local address 127.6.68.195:34343)
0503 08:13:43.048081 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:43.048083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:43.048090 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:13:43.048113 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:43.048113 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:43.048124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:43.048168 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:43.048528 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.049245 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:43.051281 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:43.051457 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:43.051499 (+ 42us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:43.051528 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:43.051576 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:43.052479 (+ 903us) server_negotiation.cc:299] Negotiation successful
0503 08:13:43.052516 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20240503 08:13:43.052897 8711 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20240503 08:13:44.578604 8170 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36500:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:44.578732 8170 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36500:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:44.579733 8170 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079: Sending DeleteTablet for 3 replicas of tablet c858a001fa994b73b9bb8ed99e51724a
I20240503 08:13:44.580022 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet c858a001fa994b73b9bb8ed99e51724a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:44 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:48118
I20240503 08:13:44.580039 8450 tablet_service.cc:1514] Processing DeleteTablet for tablet c858a001fa994b73b9bb8ed99e51724a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:44 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:44352
I20240503 08:13:44.580085 8591 tablet_service.cc:1514] Processing DeleteTablet for tablet c858a001fa994b73b9bb8ed99e51724a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:44 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:48060
I20240503 08:13:44.582458 8770 tablet_replica.cc:330] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f: stopping tablet replica
I20240503 08:13:44.582542 8770 raft_consensus.cc:2229] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:44.582594 8770 raft_consensus.cc:2258] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:44.582715 8769 tablet_replica.cc:330] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd: stopping tablet replica
I20240503 08:13:44.582881 8771 tablet_replica.cc:330] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107: stopping tablet replica
I20240503 08:13:44.582932 8770 ts_tablet_manager.cc:1875] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:44.582988 8771 raft_consensus.cc:2229] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:44.583096 8771 raft_consensus.cc:2258] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:44.583199 8769 raft_consensus.cc:2229] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:44.583288 8769 raft_consensus.cc:2258] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:44.583583 8769 ts_tablet_manager.cc:1875] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:44.583905 8771 ts_tablet_manager.cc:1875] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:44.585084 8770 ts_tablet_manager.cc:1888] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240503 08:13:44.585155 8770 log.cc:1196] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/c858a001fa994b73b9bb8ed99e51724a
I20240503 08:13:44.585422 8770 ts_tablet_manager.cc:1909] T c858a001fa994b73b9bb8ed99e51724a P 17b6072e21c84f568a1d5fc931e1884f: Deleting consensus metadata
I20240503 08:13:44.585628 8769 ts_tablet_manager.cc:1888] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240503 08:13:44.585690 8769 log.cc:1196] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/c858a001fa994b73b9bb8ed99e51724a
I20240503 08:13:44.585850 8157 catalog_manager.cc:4830] TS 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791): tablet c858a001fa994b73b9bb8ed99e51724a (table test-table [id=172a522e7fe44ad1897feaa46215e3b6]) successfully deleted
I20240503 08:13:44.585959 8769 ts_tablet_manager.cc:1909] T c858a001fa994b73b9bb8ed99e51724a P 529d5e1c5824400bb88f611330d183bd: Deleting consensus metadata
I20240503 08:13:44.586648 8157 catalog_manager.cc:4830] TS 529d5e1c5824400bb88f611330d183bd (127.6.68.194:37789): tablet c858a001fa994b73b9bb8ed99e51724a (table test-table [id=172a522e7fe44ad1897feaa46215e3b6]) successfully deleted
I20240503 08:13:44.587101 8771 ts_tablet_manager.cc:1888] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240503 08:13:44.587157 8771 log.cc:1196] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/c858a001fa994b73b9bb8ed99e51724a
I20240503 08:13:44.587431 8771 ts_tablet_manager.cc:1909] T c858a001fa994b73b9bb8ed99e51724a P 5caf60ab37dd48bfae8ab00a9208a107: Deleting consensus metadata
I20240503 08:13:44.587817 8155 catalog_manager.cc:4830] TS 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343): tablet c858a001fa994b73b9bb8ed99e51724a (table test-table [id=172a522e7fe44ad1897feaa46215e3b6]) successfully deleted
May 03 08:13:44 dist-test-slave-3g67 krb5kdc[8126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724024, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 03 08:13:44 dist-test-slave-3g67 krb5kdc[8126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724024, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:44.604643 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:44.597563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.6.68.254:34345)
0503 08:13:44.597727 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:44.597729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:44.597743 (+ 14us) server_negotiation.cc:407] Connection header received
0503 08:13:44.597776 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:44.597777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:44.597789 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:44.597848 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:44.598631 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:44.599154 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:44.599751 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:44.599944 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:44.602106 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:44.602114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:44.602114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:44.602121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:44.603500 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:44.603929 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:44.603932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:44.603932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:44.603957 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:44.604103 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:44.604104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:44.604104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:44.604205 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:44.604303 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:44.604416 (+ 113us) server_negotiation.cc:299] Negotiation successful
0503 08:13:44.604521 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20240503 08:13:44.606370 8170 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:13:44.606539 8170 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:13:44.608043 8450 tablet_service.cc:1467] Processing CreateTablet for tablet 1a64598758104212b247ec40f1661dd9 (DEFAULT_TABLE table=test-table [id=c5e4b6dc7b9647098ac02b65320807e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:44.608043 8591 tablet_service.cc:1467] Processing CreateTablet for tablet 1a64598758104212b247ec40f1661dd9 (DEFAULT_TABLE table=test-table [id=c5e4b6dc7b9647098ac02b65320807e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:44.608043 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 1a64598758104212b247ec40f1661dd9 (DEFAULT_TABLE table=test-table [id=c5e4b6dc7b9647098ac02b65320807e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:44.608193 8591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a64598758104212b247ec40f1661dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:44.608196 8450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a64598758104212b247ec40f1661dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:44.608193 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a64598758104212b247ec40f1661dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:44.609440 8780 tablet_bootstrap.cc:492] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd: Bootstrap starting.
I20240503 08:13:44.609586 8781 tablet_bootstrap.cc:492] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107: Bootstrap starting.
I20240503 08:13:44.609809 8782 tablet_bootstrap.cc:492] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f: Bootstrap starting.
I20240503 08:13:44.610101 8780 tablet_bootstrap.cc:654] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:44.610121 8781 tablet_bootstrap.cc:654] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:44.610236 8782 tablet_bootstrap.cc:654] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:44.610759 8781 tablet_bootstrap.cc:492] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107: No bootstrap required, opened a new log
I20240503 08:13:44.610821 8781 ts_tablet_manager.cc:1367] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:44.610906 8782 tablet_bootstrap.cc:492] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f: No bootstrap required, opened a new log
I20240503 08:13:44.610960 8782 ts_tablet_manager.cc:1367] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:44.610950 8781 raft_consensus.cc:348] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.611028 8781 raft_consensus.cc:374] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:44.611056 8781 raft_consensus.cc:729] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5caf60ab37dd48bfae8ab00a9208a107, State: Initialized, Role: FOLLOWER
I20240503 08:13:44.611078 8782 raft_consensus.cc:348] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.611107 8781 consensus_queue.cc:260] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.611150 8782 raft_consensus.cc:374] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:44.611169 8782 raft_consensus.cc:729] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b6072e21c84f568a1d5fc931e1884f, State: Initialized, Role: FOLLOWER
I20240503 08:13:44.611218 8781 ts_tablet_manager.cc:1398] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:44.611225 8782 consensus_queue.cc:260] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.611408 8782 ts_tablet_manager.cc:1398] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:44.611713 8780 tablet_bootstrap.cc:492] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd: No bootstrap required, opened a new log
I20240503 08:13:44.611778 8780 ts_tablet_manager.cc:1367] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:44.611910 8780 raft_consensus.cc:348] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.611982 8780 raft_consensus.cc:374] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:44.612008 8780 raft_consensus.cc:729] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529d5e1c5824400bb88f611330d183bd, State: Initialized, Role: FOLLOWER
I20240503 08:13:44.612061 8780 consensus_queue.cc:260] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.612167 8780 ts_tablet_manager.cc:1398] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:44.633396 8755 raft_consensus.cc:482] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:44.633498 8755 raft_consensus.cc:504] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.633667 8755 leader_election.cc:288] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791), 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343)
I20240503 08:13:44.633879 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a64598758104212b247ec40f1661dd9" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5caf60ab37dd48bfae8ab00a9208a107" is_pre_election: true
I20240503 08:13:44.633883 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a64598758104212b247ec40f1661dd9" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b6072e21c84f568a1d5fc931e1884f" is_pre_election: true
I20240503 08:13:44.633965 8611 raft_consensus.cc:2454] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 0.
I20240503 08:13:44.633987 8329 raft_consensus.cc:2454] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 0.
I20240503 08:13:44.634127 8401 leader_election.cc:302] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529d5e1c5824400bb88f611330d183bd, 5caf60ab37dd48bfae8ab00a9208a107; no voters:
I20240503 08:13:44.634228 8755 raft_consensus.cc:2781] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:44.634279 8755 raft_consensus.cc:482] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:44.634306 8755 raft_consensus.cc:3037] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:44.635044 8755 raft_consensus.cc:504] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.635190 8755 leader_election.cc:288] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 election: Requested vote from peers 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791), 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343)
I20240503 08:13:44.635366 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a64598758104212b247ec40f1661dd9" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b6072e21c84f568a1d5fc931e1884f"
I20240503 08:13:44.635378 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a64598758104212b247ec40f1661dd9" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5caf60ab37dd48bfae8ab00a9208a107"
I20240503 08:13:44.635450 8329 raft_consensus.cc:3037] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:44.635450 8611 raft_consensus.cc:3037] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:44.636235 8611 raft_consensus.cc:2454] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 1.
I20240503 08:13:44.636235 8329 raft_consensus.cc:2454] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 1.
I20240503 08:13:44.636479 8401 leader_election.cc:302] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529d5e1c5824400bb88f611330d183bd, 5caf60ab37dd48bfae8ab00a9208a107; no voters:
I20240503 08:13:44.636600 8755 raft_consensus.cc:2781] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:44.636677 8755 raft_consensus.cc:686] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 1 LEADER]: Becoming Leader. State: Replica: 529d5e1c5824400bb88f611330d183bd, State: Running, Role: LEADER
I20240503 08:13:44.636746 8755 consensus_queue.cc:237] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.637481 8167 catalog_manager.cc:5484] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 529d5e1c5824400bb88f611330d183bd (127.6.68.194). New cstate: current_term: 1 leader_uuid: "529d5e1c5824400bb88f611330d183bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:44.644379 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:44.642289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.6.68.194:37789)
0503 08:13:44.642416 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:44.642417 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:44.642427 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:13:44.642446 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:44.642446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:44.642455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:44.642509 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:44.642812 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:44.643314 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:44.643968 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:44.644093 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:44.644109 (+ 16us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:44.644132 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:44.644178 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:44.644274 (+ 96us) server_negotiation.cc:299] Negotiation successful
0503 08:13:44.644294 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20240503 08:13:44.648475 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:44.646674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:45035 (local address 127.6.68.194:37789)
0503 08:13:44.646752 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:44.646753 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:44.646761 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:13:44.646778 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:44.646778 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:44.646789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:44.646837 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:44.647052 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:44.647495 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:44.648059 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:44.648208 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:44.648227 (+ 19us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:44.648252 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:44.648302 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:44.648385 (+ 83us) server_negotiation.cc:299] Negotiation successful
0503 08:13:44.648404 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20240503 08:13:44.648481 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:44.646506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:37789 (local address 127.6.68.194:45035)
0503 08:13:44.646622 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:44.646634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:44.646681 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:44.646844 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:44.646845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:44.646847 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0503 08:13:44.646996 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:44.646997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:44.647551 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:44.647551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:44.647969 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:44.647971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:44.648029 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:44.648034 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0503 08:13:44.648308 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0503 08:13:44.648309 (+ 1us) client_negotiation.cc:769] Sending connection context
0503 08:13:44.648380 (+ 71us) client_negotiation.cc:240] Negotiation successful
0503 08:13:44.648417 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20240503 08:13:44.651038 8611 raft_consensus.cc:1261] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Refusing update from remote peer 529d5e1c5824400bb88f611330d183bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:44.651005 8329 raft_consensus.cc:1261] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Refusing update from remote peer 529d5e1c5824400bb88f611330d183bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:44.651286 8750 consensus_queue.cc:1035] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:44.651407 8764 consensus_queue.cc:1035] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:44.662206 8167 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36516:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:44.662340 8167 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36516:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:44.663033 8167 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079: Sending DeleteTablet for 3 replicas of tablet 1a64598758104212b247ec40f1661dd9
I20240503 08:13:44.663314 8450 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a64598758104212b247ec40f1661dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:44 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:44352
I20240503 08:13:44.663327 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a64598758104212b247ec40f1661dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:44 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:48118
I20240503 08:13:44.663354 8591 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a64598758104212b247ec40f1661dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:44 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:48060
I20240503 08:13:44.663414 8769 tablet_replica.cc:330] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd: stopping tablet replica
I20240503 08:13:44.663419 8770 tablet_replica.cc:330] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f: stopping tablet replica
I20240503 08:13:44.663446 8771 tablet_replica.cc:330] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107: stopping tablet replica
I20240503 08:13:44.663461 8769 raft_consensus.cc:2229] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:44.663465 8770 raft_consensus.cc:2229] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:44.663487 8771 raft_consensus.cc:2229] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:44.663509 8770 raft_consensus.cc:2258] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:44.663522 8771 raft_consensus.cc:2258] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:44.663547 8769 raft_consensus.cc:2258] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:44.663898 8771 ts_tablet_manager.cc:1875] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:44.663918 8167 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:13:44.664054 8167 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:13:44.664104 8769 ts_tablet_manager.cc:1875] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:44.665391 8450 tablet_service.cc:1467] Processing CreateTablet for tablet ae8d183012844fff996c434049e55bd9 (DEFAULT_TABLE table=test-table [id=1a8e22004d3b40b6906a959e6664da11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:44.665499 8450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8d183012844fff996c434049e55bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:44.665939 8770 ts_tablet_manager.cc:1875] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:44.665987 8591 tablet_service.cc:1467] Processing CreateTablet for tablet ae8d183012844fff996c434049e55bd9 (DEFAULT_TABLE table=test-table [id=1a8e22004d3b40b6906a959e6664da11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:44.666044 8769 ts_tablet_manager.cc:1888] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:13:44.666097 8769 log.cc:1196] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/1a64598758104212b247ec40f1661dd9
I20240503 08:13:44.666312 8769 ts_tablet_manager.cc:1909] T 1a64598758104212b247ec40f1661dd9 P 529d5e1c5824400bb88f611330d183bd: Deleting consensus metadata
I20240503 08:13:44.666642 8771 ts_tablet_manager.cc:1888] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:13:44.666695 8771 log.cc:1196] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/1a64598758104212b247ec40f1661dd9
I20240503 08:13:44.666900 8771 ts_tablet_manager.cc:1909] T 1a64598758104212b247ec40f1661dd9 P 5caf60ab37dd48bfae8ab00a9208a107: Deleting consensus metadata
I20240503 08:13:44.667268 8780 tablet_bootstrap.cc:492] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd: Bootstrap starting.
I20240503 08:13:44.667366 8591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8d183012844fff996c434049e55bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:44.667490 8770 ts_tablet_manager.cc:1888] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:13:44.667549 8770 log.cc:1196] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/1a64598758104212b247ec40f1661dd9
I20240503 08:13:44.667608 8780 tablet_bootstrap.cc:654] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:44.667799 8770 ts_tablet_manager.cc:1909] T 1a64598758104212b247ec40f1661dd9 P 17b6072e21c84f568a1d5fc931e1884f: Deleting consensus metadata
I20240503 08:13:44.668203 8780 tablet_bootstrap.cc:492] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd: No bootstrap required, opened a new log
I20240503 08:13:44.668246 8780 ts_tablet_manager.cc:1367] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:44.668352 8780 raft_consensus.cc:348] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.668426 8780 raft_consensus.cc:374] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:44.668449 8780 raft_consensus.cc:729] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529d5e1c5824400bb88f611330d183bd, State: Initialized, Role: FOLLOWER
I20240503 08:13:44.668502 8780 consensus_queue.cc:260] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.668558 8781 tablet_bootstrap.cc:492] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107: Bootstrap starting.
I20240503 08:13:44.668596 8780 ts_tablet_manager.cc:1398] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:44.668900 8781 tablet_bootstrap.cc:654] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:44.669166 8157 catalog_manager.cc:4830] TS 529d5e1c5824400bb88f611330d183bd (127.6.68.194:37789): tablet 1a64598758104212b247ec40f1661dd9 (table test-table [id=c5e4b6dc7b9647098ac02b65320807e3]) successfully deleted
I20240503 08:13:44.669411 8781 tablet_bootstrap.cc:492] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107: No bootstrap required, opened a new log
I20240503 08:13:44.669457 8781 ts_tablet_manager.cc:1367] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:44.669622 8781 raft_consensus.cc:348] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.669685 8781 raft_consensus.cc:374] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:44.669700 8781 raft_consensus.cc:729] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5caf60ab37dd48bfae8ab00a9208a107, State: Initialized, Role: FOLLOWER
I20240503 08:13:44.669750 8781 consensus_queue.cc:260] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.669862 8781 ts_tablet_manager.cc:1398] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:44.670377 8157 catalog_manager.cc:4830] TS 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791): tablet 1a64598758104212b247ec40f1661dd9 (table test-table [id=c5e4b6dc7b9647098ac02b65320807e3]) successfully deleted
I20240503 08:13:44.670378 8309 tablet_service.cc:1467] Processing CreateTablet for tablet ae8d183012844fff996c434049e55bd9 (DEFAULT_TABLE table=test-table [id=1a8e22004d3b40b6906a959e6664da11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:44.670498 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8d183012844fff996c434049e55bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:44.670691 8155 catalog_manager.cc:4830] TS 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343): tablet 1a64598758104212b247ec40f1661dd9 (table test-table [id=c5e4b6dc7b9647098ac02b65320807e3]) successfully deleted
I20240503 08:13:44.671849 8782 tablet_bootstrap.cc:492] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f: Bootstrap starting.
I20240503 08:13:44.672283 8782 tablet_bootstrap.cc:654] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:44.672847 8782 tablet_bootstrap.cc:492] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f: No bootstrap required, opened a new log
I20240503 08:13:44.672899 8782 ts_tablet_manager.cc:1367] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:44.673023 8782 raft_consensus.cc:348] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.673087 8782 raft_consensus.cc:374] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:44.673110 8782 raft_consensus.cc:729] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b6072e21c84f568a1d5fc931e1884f, State: Initialized, Role: FOLLOWER
I20240503 08:13:44.673161 8782 consensus_queue.cc:260] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.673262 8782 ts_tablet_manager.cc:1398] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:44.716920 8686 raft_consensus.cc:482] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:44.717027 8686 raft_consensus.cc:504] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.717185 8686 leader_election.cc:288] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791), 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343)
I20240503 08:13:44.717446 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8d183012844fff996c434049e55bd9" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b6072e21c84f568a1d5fc931e1884f" is_pre_election: true
I20240503 08:13:44.717447 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8d183012844fff996c434049e55bd9" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5caf60ab37dd48bfae8ab00a9208a107" is_pre_election: true
I20240503 08:13:44.717550 8329 raft_consensus.cc:2454] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 0.
I20240503 08:13:44.717554 8611 raft_consensus.cc:2454] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 0.
I20240503 08:13:44.717713 8401 leader_election.cc:302] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529d5e1c5824400bb88f611330d183bd, 5caf60ab37dd48bfae8ab00a9208a107; no voters:
I20240503 08:13:44.717840 8686 raft_consensus.cc:2781] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:44.717908 8686 raft_consensus.cc:482] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:44.717934 8686 raft_consensus.cc:3037] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:44.718669 8686 raft_consensus.cc:504] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.718797 8686 leader_election.cc:288] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 election: Requested vote from peers 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791), 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343)
I20240503 08:13:44.718971 8611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8d183012844fff996c434049e55bd9" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5caf60ab37dd48bfae8ab00a9208a107"
I20240503 08:13:44.718984 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8d183012844fff996c434049e55bd9" candidate_uuid: "529d5e1c5824400bb88f611330d183bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b6072e21c84f568a1d5fc931e1884f"
I20240503 08:13:44.719058 8611 raft_consensus.cc:3037] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:44.719060 8329 raft_consensus.cc:3037] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:44.719713 8329 raft_consensus.cc:2454] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 1.
I20240503 08:13:44.719949 8403 leader_election.cc:302] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b6072e21c84f568a1d5fc931e1884f, 529d5e1c5824400bb88f611330d183bd; no voters:
I20240503 08:13:44.719978 8611 raft_consensus.cc:2454] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529d5e1c5824400bb88f611330d183bd in term 1.
I20240503 08:13:44.720059 8686 raft_consensus.cc:2781] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:44.720110 8686 raft_consensus.cc:686] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 1 LEADER]: Becoming Leader. State: Replica: 529d5e1c5824400bb88f611330d183bd, State: Running, Role: LEADER
I20240503 08:13:44.720170 8686 consensus_queue.cc:237] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } }
I20240503 08:13:44.720810 8167 catalog_manager.cc:5484] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 529d5e1c5824400bb88f611330d183bd (127.6.68.194). New cstate: current_term: 1 leader_uuid: "529d5e1c5824400bb88f611330d183bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "529d5e1c5824400bb88f611330d183bd" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 37789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:44.735883 8329 raft_consensus.cc:1261] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Refusing update from remote peer 529d5e1c5824400bb88f611330d183bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:44.736029 8611 raft_consensus.cc:1261] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Refusing update from remote peer 529d5e1c5824400bb88f611330d183bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:44.736100 8686 consensus_queue.cc:1035] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b6072e21c84f568a1d5fc931e1884f" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:44.736280 8686 consensus_queue.cc:1035] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5caf60ab37dd48bfae8ab00a9208a107" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:44.740944 8748 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20240503 08:13:46.263520 8167 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36516:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:46.263659 8167 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36516:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:46.264604 8167 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 690762e09f4b4777bc27f56ef249a079: Sending DeleteTablet for 3 replicas of tablet ae8d183012844fff996c434049e55bd9
I20240503 08:13:46.264902 8449 tablet_service.cc:1514] Processing DeleteTablet for tablet ae8d183012844fff996c434049e55bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:46 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:44352
I20240503 08:13:46.264889 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet ae8d183012844fff996c434049e55bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:46 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:48118
I20240503 08:13:46.264971 8591 tablet_service.cc:1514] Processing DeleteTablet for tablet ae8d183012844fff996c434049e55bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:46 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:48060
I20240503 08:13:46.265141 8824 tablet_replica.cc:330] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f: stopping tablet replica
I20240503 08:13:46.265175 8823 tablet_replica.cc:330] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd: stopping tablet replica
I20240503 08:13:46.265203 8824 raft_consensus.cc:2229] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:46.265233 8823 raft_consensus.cc:2229] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:46.265266 8824 raft_consensus.cc:2258] T ae8d183012844fff996c434049e55bd9 P 17b6072e21c84f568a1d5fc931e1884f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:46.265283 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 8244
I20240503 08:13:46.265329 8823 raft_consensus.cc:2258] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:46.266064 8825 tablet_replica.cc:330] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107: stopping tablet replica
I20240503 08:13:46.266134 8825 raft_consensus.cc:2229] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:46.266191 8825 raft_consensus.cc:2258] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:46.266242 8823 ts_tablet_manager.cc:1875] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:46.266490 8825 ts_tablet_manager.cc:1875] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:46.268113 8823 ts_tablet_manager.cc:1888] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240503 08:13:46.268252 8823 log.cc:1196] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/ae8d183012844fff996c434049e55bd9
I20240503 08:13:46.268604 8823 ts_tablet_manager.cc:1909] T ae8d183012844fff996c434049e55bd9 P 529d5e1c5824400bb88f611330d183bd: Deleting consensus metadata
I20240503 08:13:46.268623 8825 ts_tablet_manager.cc:1888] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240503 08:13:46.268677 8825 log.cc:1196] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/ae8d183012844fff996c434049e55bd9
I20240503 08:13:46.268929 8825 ts_tablet_manager.cc:1909] T ae8d183012844fff996c434049e55bd9 P 5caf60ab37dd48bfae8ab00a9208a107: Deleting consensus metadata
I20240503 08:13:46.271826 8157 catalog_manager.cc:4830] TS 529d5e1c5824400bb88f611330d183bd (127.6.68.194:37789): tablet ae8d183012844fff996c434049e55bd9 (table test-table [id=1a8e22004d3b40b6906a959e6664da11]) successfully deleted
I20240503 08:13:46.272200 8155 catalog_manager.cc:4830] TS 5caf60ab37dd48bfae8ab00a9208a107 (127.6.68.195:34343): tablet ae8d183012844fff996c434049e55bd9 (table test-table [id=1a8e22004d3b40b6906a959e6664da11]) successfully deleted
W20240503 08:13:46.272897 8157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:39791 (error 108)
I20240503 08:13:46.273161 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 8385
I20240503 08:13:46.273931 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:46.273321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39791 (local address 127.0.0.1:48126)
0503 08:13:46.273832 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:46.273878 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:39791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":60,"threads_started":1}
W20240503 08:13:46.274047 8157 catalog_manager.cc:4573] TS 17b6072e21c84f568a1d5fc931e1884f (127.6.68.193:39791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae8d183012844fff996c434049e55bd9: Network error: Client connection negotiation failed: client connection to 127.6.68.193:39791: connect: Connection refused (error 111)
I20240503 08:13:46.279790 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 8525
I20240503 08:13:46.285928 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 8139
2024-05-03T08:13:46Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5482 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:13:46 dist-test-slave-3g67 krb5kdc[8830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:13:46 dist-test-slave-3g67 krb5kdc[8830](info): set up 2 sockets
May 03 08:13:46 dist-test-slave-3g67 krb5kdc[8830](info): commencing operation
krb5kdc: starting...
W20240503 08:13:47.252007 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:13:47 dist-test-slave-3g67 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:13:47.371882 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34805
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:34805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:47.452242 8843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:47.452447 8843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:47.452469 8843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:47.453752 8843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:47.453799 8843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:47.453814 8843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:47.453824 8843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:47.453836 8843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:47.455440 8843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:34805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:47.455700 8843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:47.455897 8843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:47.460517 8848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.461167 8851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.462072 8849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:47.462895 8843 server_base.cc:1034] running on GCE node
I20240503 08:13:47.463138 8843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:47.463407 8843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:47.464565 8843 hybrid_clock.cc:648] HybridClock initialized: now 1714724027464524 us; error 71 us; skew 500 ppm
May 03 08:13:47 dist-test-slave-3g67 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:47.467290 8843 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:13:47.467773 8843 webserver.cc:460] Webserver started at http://127.6.68.254:39595/ using document root <none> and password file <none>
I20240503 08:13:47.467979 8843 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:47.468026 8843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:47.468142 8843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:47.469166 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "e8dbf00778cc4113b8c9dcb25347aaa5"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "d9959a1befcfa620113a4deacdc84b7c"
server_key_iv: "b55e4fa94ae3937a5fbe9efb51c289a1"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.469496 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "e8dbf00778cc4113b8c9dcb25347aaa5"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "d9959a1befcfa620113a4deacdc84b7c"
server_key_iv: "b55e4fa94ae3937a5fbe9efb51c289a1"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.470824 8843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:13:47.471711 8858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.471910 8843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:47.472000 8843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "e8dbf00778cc4113b8c9dcb25347aaa5"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "d9959a1befcfa620113a4deacdc84b7c"
server_key_iv: "b55e4fa94ae3937a5fbe9efb51c289a1"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.472070 8843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:47.506124 8843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:47.508322 8843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:47.508466 8843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:47.512326 8843 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:34805
I20240503 08:13:47.512369 8910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:34805 every 8 connection(s)
I20240503 08:13:47.512696 8843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:13:47.513310 8911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:47.515911 8911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5: Bootstrap starting.
I20240503 08:13:47.516487 8911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:47.516752 8911 log.cc:824] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:47.516896 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 8843
I20240503 08:13:47.516993 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:13:47.517156 6419 external_mini_cluster.cc:1428] Setting key f3bfb031c5e58c0a3b1067c0e7e26156
I20240503 08:13:47.517526 8911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5: No bootstrap required, opened a new log
I20240503 08:13:47.519210 8911 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dbf00778cc4113b8c9dcb25347aaa5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34805 } }
I20240503 08:13:47.519351 8911 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:47.519367 8911 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8dbf00778cc4113b8c9dcb25347aaa5, State: Initialized, Role: FOLLOWER
I20240503 08:13:47.519491 8911 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dbf00778cc4113b8c9dcb25347aaa5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34805 } }
I20240503 08:13:47.519565 8911 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:47.519599 8911 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:47.519671 8911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:47.520262 8911 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dbf00778cc4113b8c9dcb25347aaa5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34805 } }
I20240503 08:13:47.520368 8911 leader_election.cc:302] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8dbf00778cc4113b8c9dcb25347aaa5; no voters:
I20240503 08:13:47.520555 8911 leader_election.cc:288] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:47.520645 8916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:47.520793 8911 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:47.520836 8916 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [term 1 LEADER]: Becoming Leader. State: Replica: e8dbf00778cc4113b8c9dcb25347aaa5, State: Running, Role: LEADER
I20240503 08:13:47.521023 8916 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dbf00778cc4113b8c9dcb25347aaa5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34805 } }
I20240503 08:13:47.521351 8918 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8dbf00778cc4113b8c9dcb25347aaa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dbf00778cc4113b8c9dcb25347aaa5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34805 } } }
I20240503 08:13:47.521435 8918 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:47.521338 8916 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8dbf00778cc4113b8c9dcb25347aaa5. Latest consensus state: current_term: 1 leader_uuid: "e8dbf00778cc4113b8c9dcb25347aaa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8dbf00778cc4113b8c9dcb25347aaa5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34805 } } }
I20240503 08:13:47.521704 8916 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:47.521847 8924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:47.523228 8924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:47.524855 8924 catalog_manager.cc:1347] Generated new cluster ID: f5ab313b48fe4630beae207806513bc6
I20240503 08:13:47.524914 8924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 03 08:13:47 dist-test-slave-3g67 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:47.528509 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:47.517735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.6.68.254:34805)
0503 08:13:47.517921 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:47.517933 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:47.517953 (+ 20us) server_negotiation.cc:407] Connection header received
0503 08:13:47.519180 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:47.519189 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:47.519241 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:47.519404 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:47.520318 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:47.521918 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:47.522624 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:47.522895 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:47.525437 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:47.525464 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:47.525465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:47.525485 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:47.527169 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:47.527511 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:47.527514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:47.527515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:47.527561 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:47.527726 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:47.527727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:47.527727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:47.527917 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:47.528068 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:47.528183 (+ 115us) server_negotiation.cc:299] Negotiation successful
0503 08:13:47.528331 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20240503 08:13:47.538378 8924 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:47.539036 8924 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:47.542703 8924 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e8dbf00778cc4113b8c9dcb25347aaa5: Generated new TSK 0
I20240503 08:13:47.542917 8924 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:13:47.574287 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34805
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:47.656909 8937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:47.657130 8937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:47.657178 8937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:47.658566 8937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:47.658617 8937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:47.658644 8937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:13:47.660358 8937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:47.660588 8937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:47.660940 8937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:47.666318 8943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.666966 8942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.667771 8945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:47.668375 8937 server_base.cc:1034] running on GCE node
I20240503 08:13:47.668618 8937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:47.668871 8937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:47.670013 8937 hybrid_clock.cc:648] HybridClock initialized: now 1714724027669997 us; error 42 us; skew 500 ppm
May 03 08:13:47 dist-test-slave-3g67 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:47.672613 8937 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:13:47.673089 8937 webserver.cc:460] Webserver started at http://127.6.68.193:35093/ using document root <none> and password file <none>
I20240503 08:13:47.673300 8937 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:47.673343 8937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:47.673447 8937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:47.674460 8937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "340f045e0f6a40d1a9a87b9b78fffa2d"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "0244e967428080ea21bbd90e09182db2"
server_key_iv: "4e070e7e15356e5c0638cfbe2336b960"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.674758 8937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "340f045e0f6a40d1a9a87b9b78fffa2d"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "0244e967428080ea21bbd90e09182db2"
server_key_iv: "4e070e7e15356e5c0638cfbe2336b960"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.676057 8937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:13:47.676761 8952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.676919 8937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:13:47.676993 8937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "340f045e0f6a40d1a9a87b9b78fffa2d"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "0244e967428080ea21bbd90e09182db2"
server_key_iv: "4e070e7e15356e5c0638cfbe2336b960"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.677063 8937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:47.691787 8937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:47.693951 8937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:47.694063 8937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:47.694326 8937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:47.694625 8937 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:47.694658 8937 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.694681 8937 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:47.694707 8937 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.700322 8937 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:35149
I20240503 08:13:47.700393 9065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:35149 every 8 connection(s)
I20240503 08:13:47.700670 8937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
May 03 08:13:47 dist-test-slave-3g67 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:47.709422 9068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:47.701333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34805 (local address 127.6.68.193:58585)
0503 08:13:47.701478 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:47.701508 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:47.701948 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:47.702198 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:47.702202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:47.702521 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:47.702802 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:47.702806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:47.703538 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:47.703539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:47.703959 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:47.703961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:47.704327 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:47.704873 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:47.704890 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:47.706327 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:47.708166 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:47.708172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:47.708175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:47.708421 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:47.708639 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:47.708640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:47.708640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:47.708706 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:47.708942 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:47.708945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:47.709061 (+ 116us) client_negotiation.cc:769] Sending connection context
0503 08:13:47.709152 (+ 91us) client_negotiation.cc:240] Negotiation successful
0503 08:13:47.709268 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20240503 08:13:47.709411 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 8937
I20240503 08:13:47.709471 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:47.701396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:58585 (local address 127.6.68.254:34805)
0503 08:13:47.701466 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:47.701468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:47.701593 (+ 125us) server_negotiation.cc:407] Connection header received
0503 08:13:47.702041 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:47.702042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:47.702058 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:47.702142 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:47.702920 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:47.703459 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:47.704043 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:47.704182 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:47.706434 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:47.706438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:47.706438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:47.706445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:47.708082 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:47.708513 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:47.708515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:47.708515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:47.708545 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:47.708769 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:47.708769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:47.708770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:47.708887 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:47.708942 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:47.709227 (+ 285us) server_negotiation.cc:299] Negotiation successful
0503 08:13:47.709336 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240503 08:13:47.709519 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:13:47.709769 6419 external_mini_cluster.cc:1428] Setting key 286ec34d68aaaac00b91f32423320798
I20240503 08:13:47.710439 9066 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34805
I20240503 08:13:47.710561 9066 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:47.710820 9066 heartbeater.cc:511] Master 127.6.68.254:34805 requested a full tablet report, sending...
I20240503 08:13:47.711418 8875 ts_manager.cc:194] Registered new tserver with Master: 340f045e0f6a40d1a9a87b9b78fffa2d (127.6.68.193:35149)
I20240503 08:13:47.712325 8875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:58585
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:13:47.735934 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34805
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:47.816900 9071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:47.817109 9071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:47.817147 9071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:47.818459 9071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:47.818511 9071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:47.818553 9071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:13:47.820302 9071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:47.820541 9071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:47.820734 9071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:47.825786 9079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.826246 9076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.826890 9077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:47.827894 9071 server_base.cc:1034] running on GCE node
I20240503 08:13:47.828099 9071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:47.828333 9071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:47.829483 9071 hybrid_clock.cc:648] HybridClock initialized: now 1714724027829458 us; error 36 us; skew 500 ppm
May 03 08:13:47 dist-test-slave-3g67 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:47.832072 9071 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:13:47.832572 9071 webserver.cc:460] Webserver started at http://127.6.68.194:45297/ using document root <none> and password file <none>
I20240503 08:13:47.832760 9071 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:47.832808 9071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:47.832932 9071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:47.833959 9071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "2b84cc1848704a5eb84c123d55fc630d"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "c069c2eea883a545f876d5e6319de420"
server_key_iv: "0a5da350bd5d0078cfebe55e58f91d1b"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.834282 9071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "2b84cc1848704a5eb84c123d55fc630d"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "c069c2eea883a545f876d5e6319de420"
server_key_iv: "0a5da350bd5d0078cfebe55e58f91d1b"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.835598 9071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:13:47.836428 9086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.836592 9071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:47.836655 9071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "2b84cc1848704a5eb84c123d55fc630d"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "c069c2eea883a545f876d5e6319de420"
server_key_iv: "0a5da350bd5d0078cfebe55e58f91d1b"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.836725 9071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:47.847802 9071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:47.849989 9071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:47.850098 9071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:47.850325 9071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:47.850651 9071 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:47.850684 9071 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.850709 9071 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:47.850737 9071 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.856170 9071 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:42643
I20240503 08:13:47.856220 9199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:42643 every 8 connection(s)
I20240503 08:13:47.856527 9071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:13:47.860599 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 9071
I20240503 08:13:47.860707 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:13:47.860909 6419 external_mini_cluster.cc:1428] Setting key ea43e8c482a98f6fd25cffcc1bb7ce0a
May 03 08:13:47 dist-test-slave-3g67 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:47.865417 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:47.857311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:42719 (local address 127.6.68.254:34805)
0503 08:13:47.857452 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:47.857455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:47.857518 (+ 63us) server_negotiation.cc:407] Connection header received
0503 08:13:47.857992 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:47.857993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:47.858002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:47.858071 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:47.858863 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:47.859358 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:47.860030 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:47.860164 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:47.862686 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:47.862690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:47.862690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:47.862697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:47.864135 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:47.864603 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:47.864603 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:47.864603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:47.864619 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:47.864806 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:47.864806 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:47.864807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:47.864889 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:47.864932 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:47.865276 (+ 344us) server_negotiation.cc:299] Negotiation successful
0503 08:13:47.865348 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20240503 08:13:47.865654 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:47.857226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34805 (local address 127.6.68.194:42719)
0503 08:13:47.857426 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:47.857453 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:47.857903 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:47.858137 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:47.858141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:47.858453 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:47.858744 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:47.858758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:47.859467 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:47.859469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:47.859933 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:47.859935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:47.860274 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:47.860919 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:47.860947 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:47.862576 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:47.864220 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:47.864225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:47.864229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:47.864530 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:47.864671 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:47.864672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:47.864672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:47.864752 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:47.864941 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:47.864945 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:47.865113 (+ 168us) client_negotiation.cc:769] Sending connection context
0503 08:13:47.865266 (+ 153us) client_negotiation.cc:240] Negotiation successful
0503 08:13:47.865434 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":120,"threads_started":1}
I20240503 08:13:47.866217 9200 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34805
I20240503 08:13:47.866328 9200 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:47.866544 9200 heartbeater.cc:511] Master 127.6.68.254:34805 requested a full tablet report, sending...
I20240503 08:13:47.866945 8875 ts_manager.cc:194] Registered new tserver with Master: 2b84cc1848704a5eb84c123d55fc630d (127.6.68.194:42643)
I20240503 08:13:47.867326 8875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:42719
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:13:47.889118 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34805
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:47.969056 9205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:47.969264 9205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:47.969295 9205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:47.970641 9205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:47.970697 9205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:47.970724 9205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:13:47.972419 9205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:47.972643 9205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:47.972849 9205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:47.977726 9213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.978552 9210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:47.979317 9211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:47.979995 9205 server_base.cc:1034] running on GCE node
I20240503 08:13:47.980180 9205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:47.980448 9205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:47.981595 9205 hybrid_clock.cc:648] HybridClock initialized: now 1714724027981576 us; error 39 us; skew 500 ppm
May 03 08:13:47 dist-test-slave-3g67 krb5kdc[8830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:47.984175 9205 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:13:47.984580 9205 webserver.cc:460] Webserver started at http://127.6.68.195:38403/ using document root <none> and password file <none>
I20240503 08:13:47.984762 9205 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:47.984803 9205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:47.984897 9205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:47.985889 9205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "7ac051c8e8a64bf7a298141176901a26"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "7fcf408ff34847930bc2075eefee05fb"
server_key_iv: "8dac0aeb0eb5d57237ae29abc87042c7"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.986196 9205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "7ac051c8e8a64bf7a298141176901a26"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "7fcf408ff34847930bc2075eefee05fb"
server_key_iv: "8dac0aeb0eb5d57237ae29abc87042c7"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.987432 9205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:13:47.988155 9220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.988317 9205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:47.988394 9205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "7ac051c8e8a64bf7a298141176901a26"
format_stamp: "Formatted at 2024-05-03 08:13:47 on dist-test-slave-3g67"
server_key: "7fcf408ff34847930bc2075eefee05fb"
server_key_iv: "8dac0aeb0eb5d57237ae29abc87042c7"
server_key_version: "encryptionkey@0"
I20240503 08:13:47.988456 9205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:48.006824 9205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:48.009232 9205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:48.009348 9205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:48.009595 9205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:48.009896 9205 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:48.009929 9205 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:48.009953 9205 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:48.009965 9205 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:48.015524 9205 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:36231
I20240503 08:13:48.015584 9333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:36231 every 8 connection(s)
I20240503 08:13:48.015901 9205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
May 03 08:13:48 dist-test-slave-3g67 krb5kdc[8830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:48.024152 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 9205
I20240503 08:13:48.024257 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:13:48.024420 6419 external_mini_cluster.cc:1428] Setting key 55e56aa5d9626db921e82d74c5c42fd1
I20240503 08:13:48.024684 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:48.016613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:36257 (local address 127.6.68.254:34805)
0503 08:13:48.016707 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:48.016709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:48.016818 (+ 109us) server_negotiation.cc:407] Connection header received
0503 08:13:48.017332 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:48.017334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:48.017348 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:48.017415 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:48.018138 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:48.018672 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:48.019228 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:48.019330 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:48.021797 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:48.021802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:48.021802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:48.021808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:48.023294 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:48.023703 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:48.023704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:48.023705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:48.023723 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:48.023932 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:48.023933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:48.023934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:48.024050 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:48.024136 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:48.024465 (+ 329us) server_negotiation.cc:299] Negotiation successful
0503 08:13:48.024587 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240503 08:13:48.024962 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:48.016578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34805 (local address 127.6.68.195:36257)
0503 08:13:48.016714 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:48.016740 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:48.017214 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:48.017465 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:48.017469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:48.017776 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:48.018057 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:48.018060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:48.018753 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:48.018754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:48.019140 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:48.019142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:48.019519 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:48.020115 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:48.020132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:48.021696 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:48.023370 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:48.023376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:48.023378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:48.023577 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:48.023773 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:48.023774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:48.023774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:48.023848 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:48.024144 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:48.024151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:48.024305 (+ 154us) client_negotiation.cc:769] Sending connection context
0503 08:13:48.024434 (+ 129us) client_negotiation.cc:240] Negotiation successful
0503 08:13:48.024709 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20240503 08:13:48.025413 9334 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34805
I20240503 08:13:48.025514 9334 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:48.025723 9334 heartbeater.cc:511] Master 127.6.68.254:34805 requested a full tablet report, sending...
I20240503 08:13:48.026075 8870 ts_manager.cc:194] Registered new tserver with Master: 7ac051c8e8a64bf7a298141176901a26 (127.6.68.195:36231)
I20240503 08:13:48.026480 8870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:36257
I20240503 08:13:48.036530 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240503 08:13:48.046679 8915 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:13:48.045782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.6.68.254:34805)
0503 08:13:48.045838 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:48.045841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:48.045949 (+ 108us) server_negotiation.cc:407] Connection header received
0503 08:13:48.045981 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:48.045982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:48.045992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:48.046057 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:48.046624 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35322: BlockingRecv error: recv got EOF from 127.0.0.1:35322 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:13:48.047849 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 8937
I20240503 08:13:48.052543 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 9071
I20240503 08:13:48.057104 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 9205
I20240503 08:13:48.061676 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 8843
2024-05-03T08:13:48Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1773 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:13:48 dist-test-slave-3g67 krb5kdc[9346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:13:48 dist-test-slave-3g67 krb5kdc[9346](info): set up 2 sockets
May 03 08:13:48 dist-test-slave-3g67 krb5kdc[9346](info): commencing operation
krb5kdc: starting...
W20240503 08:13:49.020000 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:13:49 dist-test-slave-3g67 krb5kdc[9346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:13:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:13:49.143350 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:38091
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:49.224085 9359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:49.224311 9359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:49.224335 9359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:49.225694 9359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:49.225749 9359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:49.225764 9359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:49.225776 9359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:49.225788 9359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:49.227519 9359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:49.227783 9359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:49.227962 9359 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:49.232612 9364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:49.233330 9367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:49.233893 9365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:49.234876 9359 server_base.cc:1034] running on GCE node
I20240503 08:13:49.235136 9359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:49.235489 9359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:49.236630 9359 hybrid_clock.cc:648] HybridClock initialized: now 1714724029236615 us; error 31 us; skew 500 ppm
May 03 08:13:49 dist-test-slave-3g67 krb5kdc[9346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724029, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:49.239276 9359 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:13:49.239769 9359 webserver.cc:460] Webserver started at http://127.6.68.254:34321/ using document root <none> and password file <none>
I20240503 08:13:49.239959 9359 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:49.240002 9359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:49.240113 9359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:49.241163 9359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "74a4de56dce94446b97f19a35c0e5199"
format_stamp: "Formatted at 2024-05-03 08:13:49 on dist-test-slave-3g67"
server_key: "f7ad420ae29ea6f6b8290ec0ab43c34e"
server_key_iv: "6307424959c03ec6e42223beb833a355"
server_key_version: "encryptionkey@0"
I20240503 08:13:49.241480 9359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "74a4de56dce94446b97f19a35c0e5199"
format_stamp: "Formatted at 2024-05-03 08:13:49 on dist-test-slave-3g67"
server_key: "f7ad420ae29ea6f6b8290ec0ab43c34e"
server_key_iv: "6307424959c03ec6e42223beb833a355"
server_key_version: "encryptionkey@0"
I20240503 08:13:49.242734 9359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:13:49.243444 9374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:49.243687 9359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:49.243777 9359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "74a4de56dce94446b97f19a35c0e5199"
format_stamp: "Formatted at 2024-05-03 08:13:49 on dist-test-slave-3g67"
server_key: "f7ad420ae29ea6f6b8290ec0ab43c34e"
server_key_iv: "6307424959c03ec6e42223beb833a355"
server_key_version: "encryptionkey@0"
I20240503 08:13:49.243840 9359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:49.260398 9359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:49.262902 9359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:49.263038 9359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:49.266842 9359 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:38091
I20240503 08:13:49.266880 9426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:38091 every 8 connection(s)
I20240503 08:13:49.267210 9359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:13:49.267905 9427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:49.267937 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 9359
I20240503 08:13:49.268033 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:13:49.268191 6419 external_mini_cluster.cc:1428] Setting key dd876820c8b48cdc920324ea8169e964
I20240503 08:13:49.271414 9427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199: Bootstrap starting.
I20240503 08:13:49.272045 9427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:49.272702 9427 log.cc:824] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:49.273490 9427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199: No bootstrap required, opened a new log
May 03 08:13:49 dist-test-slave-3g67 krb5kdc[9346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:49.275404 9427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4de56dce94446b97f19a35c0e5199" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38091 } }
I20240503 08:13:49.275552 9427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:49.275585 9427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4de56dce94446b97f19a35c0e5199, State: Initialized, Role: FOLLOWER
I20240503 08:13:49.275758 9427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4de56dce94446b97f19a35c0e5199" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38091 } }
I20240503 08:13:49.275842 9427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:49.275885 9427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:49.275944 9427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:49.276552 9427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4de56dce94446b97f19a35c0e5199" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38091 } }
I20240503 08:13:49.276669 9427 leader_election.cc:302] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74a4de56dce94446b97f19a35c0e5199; no voters:
I20240503 08:13:49.276856 9427 leader_election.cc:288] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:49.276909 9432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:49.277079 9432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4de56dce94446b97f19a35c0e5199, State: Running, Role: LEADER
I20240503 08:13:49.277117 9427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:49.277185 9432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4de56dce94446b97f19a35c0e5199" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38091 } }
I20240503 08:13:49.277504 9432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74a4de56dce94446b97f19a35c0e5199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4de56dce94446b97f19a35c0e5199" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38091 } } }
I20240503 08:13:49.277666 9432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:49.277865 9432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74a4de56dce94446b97f19a35c0e5199. Latest consensus state: current_term: 1 leader_uuid: "74a4de56dce94446b97f19a35c0e5199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a4de56dce94446b97f19a35c0e5199" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38091 } } }
I20240503 08:13:49.277993 9432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:49.277999 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:49.268695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.6.68.254:38091)
0503 08:13:49.268865 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:49.268875 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:49.268893 (+ 18us) server_negotiation.cc:407] Connection header received
0503 08:13:49.269632 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:49.269639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:49.269690 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:49.269858 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:49.270686 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:49.271411 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:49.272061 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:49.272321 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:49.274654 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:49.274663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:49.274663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:49.274682 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:49.276490 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:49.276859 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:49.276862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:49.276863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:49.276929 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:49.277099 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:49.277100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:49.277100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:49.277375 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:49.277550 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:49.277631 (+ 81us) server_negotiation.cc:299] Negotiation successful
0503 08:13:49.277756 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20240503 08:13:49.278191 9441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:49.278810 9441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:49.279266 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:13:49.280372 9441 catalog_manager.cc:1347] Generated new cluster ID: ca5b26d2c07f4d6f838d08776a7f51da
I20240503 08:13:49.280433 9441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:13:49.294550 9441 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:49.295276 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:49.291364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.6.68.254:38091)
0503 08:13:49.291416 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:49.291419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:49.291519 (+ 100us) server_negotiation.cc:407] Connection header received
0503 08:13:49.291586 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:49.291587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:49.291599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:49.291713 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:49.292457 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:49.293191 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:49.294015 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:49.294183 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:49.294772 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:49.294777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:49.294777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:49.294977 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:49.294993 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:49.294998 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:49.295053 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:49.295136 (+ 83us) server_negotiation.cc:299] Negotiation successful
0503 08:13:49.295174 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240503 08:13:49.295334 9441 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:49.306056 9441 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 74a4de56dce94446b97f19a35c0e5199: Generated new TSK 0
I20240503 08:13:49.306277 9441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:13:49.309520 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 9359
2024-05-03T08:13:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1246 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:13:49 dist-test-slave-3g67 krb5kdc[9460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:13:49 dist-test-slave-3g67 krb5kdc[9460](info): set up 2 sockets
May 03 08:13:49 dist-test-slave-3g67 krb5kdc[9460](info): commencing operation
krb5kdc: starting...
W20240503 08:13:50.284123 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:13:50 dist-test-slave-3g67 krb5kdc[9460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:13:50.407361 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:43753
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:43753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:50.489259 9473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:50.489487 9473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:50.489512 9473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:50.490911 9473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:50.490974 9473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:50.490988 9473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:50.491000 9473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:50.491012 9473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:50.492762 9473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:43753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:43753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:50.492987 9473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:50.493196 9473 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:50.498399 9481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:50.498816 9479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:50.499497 9478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:50.500497 9473 server_base.cc:1034] running on GCE node
I20240503 08:13:50.500756 9473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:50.501014 9473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:50.502153 9473 hybrid_clock.cc:648] HybridClock initialized: now 1714724030502140 us; error 32 us; skew 500 ppm
May 03 08:13:50 dist-test-slave-3g67 krb5kdc[9460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724030, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:50.504791 9473 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:13:50.505218 9473 webserver.cc:460] Webserver started at http://127.6.68.254:42549/ using document root <none> and password file <none>
I20240503 08:13:50.505415 9473 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:50.505458 9473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:50.505561 9473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:50.506629 9473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "63bd963b8f3e4119b3d95c98b2a796fd"
format_stamp: "Formatted at 2024-05-03 08:13:50 on dist-test-slave-3g67"
server_key: "3d9cc1e0a6f29f9a8618afc543413079"
server_key_iv: "f03a1ff0a5a1d3fd9e1b86f7c0a80104"
server_key_version: "encryptionkey@0"
I20240503 08:13:50.506956 9473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "63bd963b8f3e4119b3d95c98b2a796fd"
format_stamp: "Formatted at 2024-05-03 08:13:50 on dist-test-slave-3g67"
server_key: "3d9cc1e0a6f29f9a8618afc543413079"
server_key_iv: "f03a1ff0a5a1d3fd9e1b86f7c0a80104"
server_key_version: "encryptionkey@0"
I20240503 08:13:50.508281 9473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:13:50.509016 9488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:50.509184 9473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:50.509268 9473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "63bd963b8f3e4119b3d95c98b2a796fd"
format_stamp: "Formatted at 2024-05-03 08:13:50 on dist-test-slave-3g67"
server_key: "3d9cc1e0a6f29f9a8618afc543413079"
server_key_iv: "f03a1ff0a5a1d3fd9e1b86f7c0a80104"
server_key_version: "encryptionkey@0"
I20240503 08:13:50.509336 9473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:50.541002 9473 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:50.543226 9473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:50.543398 9473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:50.547300 9473 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:43753
I20240503 08:13:50.547331 9541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:43753 every 8 connection(s)
I20240503 08:13:50.547684 9473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:13:50.548585 9542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:50.550818 9542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd: Bootstrap starting.
I20240503 08:13:50.551404 9542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:50.551689 9542 log.cc:824] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:50.552459 9542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd: No bootstrap required, opened a new log
I20240503 08:13:50.552491 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 9473
I20240503 08:13:50.552580 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:13:50.552771 6419 external_mini_cluster.cc:1428] Setting key 17b6ebca8cd8b5b0ac3285ef696b1a53
I20240503 08:13:50.554159 9542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bd963b8f3e4119b3d95c98b2a796fd" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43753 } }
I20240503 08:13:50.554309 9542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:50.554339 9542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63bd963b8f3e4119b3d95c98b2a796fd, State: Initialized, Role: FOLLOWER
I20240503 08:13:50.554447 9542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bd963b8f3e4119b3d95c98b2a796fd" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43753 } }
I20240503 08:13:50.554526 9542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:50.554558 9542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:50.554612 9542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:50.555361 9542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bd963b8f3e4119b3d95c98b2a796fd" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43753 } }
I20240503 08:13:50.555464 9542 leader_election.cc:302] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63bd963b8f3e4119b3d95c98b2a796fd; no voters:
I20240503 08:13:50.555666 9542 leader_election.cc:288] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:50.555720 9547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:50.555866 9547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [term 1 LEADER]: Becoming Leader. State: Replica: 63bd963b8f3e4119b3d95c98b2a796fd, State: Running, Role: LEADER
I20240503 08:13:50.555932 9542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:50.556010 9547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bd963b8f3e4119b3d95c98b2a796fd" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43753 } }
I20240503 08:13:50.556447 9548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63bd963b8f3e4119b3d95c98b2a796fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bd963b8f3e4119b3d95c98b2a796fd" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43753 } } }
I20240503 08:13:50.556551 9548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:50.556748 9549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63bd963b8f3e4119b3d95c98b2a796fd. Latest consensus state: current_term: 1 leader_uuid: "63bd963b8f3e4119b3d95c98b2a796fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bd963b8f3e4119b3d95c98b2a796fd" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43753 } } }
I20240503 08:13:50.556869 9549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:50.557021 9555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:50.557541 9555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:50.559127 9555 catalog_manager.cc:1347] Generated new cluster ID: 0d55986977f0423fb092896768daf15c
I20240503 08:13:50.559186 9555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 03 08:13:50 dist-test-slave-3g67 krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:50.564253 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:50.553303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.6.68.254:43753)
0503 08:13:50.553466 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:50.553476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:50.553494 (+ 18us) server_negotiation.cc:407] Connection header received
0503 08:13:50.554122 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:50.554128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:50.554168 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:50.554290 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:50.555270 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:50.556678 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:50.557722 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:50.557990 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:50.560396 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:50.560416 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:50.560417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:50.560431 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:50.562616 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:50.563034 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:50.563036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:50.563037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:50.563102 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:50.563250 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:50.563251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:50.563251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:50.563505 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:50.563679 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:50.563787 (+ 108us) server_negotiation.cc:299] Negotiation successful
0503 08:13:50.564045 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20240503 08:13:50.571265 9555 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:50.572072 9555 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:50.587141 9555 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 63bd963b8f3e4119b3d95c98b2a796fd: Generated new TSK 0
I20240503 08:13:50.587365 9555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:13:50.623724 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43753
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:50.710414 9568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:50.710644 9568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:50.710669 9568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:50.712114 9568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:50.712177 9568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:50.712206 9568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:13:50.713847 9568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:50.714071 9568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:50.714289 9568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:50.719808 9573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:50.720700 9574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:50.721336 9576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:50.722023 9568 server_base.cc:1034] running on GCE node
I20240503 08:13:50.722224 9568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:50.722478 9568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:50.723652 9568 hybrid_clock.cc:648] HybridClock initialized: now 1714724030723634 us; error 38 us; skew 500 ppm
May 03 08:13:50 dist-test-slave-3g67 krb5kdc[9460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724030, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:50.726455 9568 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:13:50.726958 9568 webserver.cc:460] Webserver started at http://127.6.68.193:44837/ using document root <none> and password file <none>
I20240503 08:13:50.727152 9568 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:50.727198 9568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:50.727316 9568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:50.728403 9568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "e0ce160a09b1462a91c99d4b674c713a"
format_stamp: "Formatted at 2024-05-03 08:13:50 on dist-test-slave-3g67"
server_key: "380495656f9f4e70db89037d299f1da5"
server_key_iv: "360a364013c1f7d7087cc703e7932029"
server_key_version: "encryptionkey@0"
I20240503 08:13:50.728713 9568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "e0ce160a09b1462a91c99d4b674c713a"
format_stamp: "Formatted at 2024-05-03 08:13:50 on dist-test-slave-3g67"
server_key: "380495656f9f4e70db89037d299f1da5"
server_key_iv: "360a364013c1f7d7087cc703e7932029"
server_key_version: "encryptionkey@0"
I20240503 08:13:50.729956 9568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:13:50.730628 9583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:50.730862 9568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:50.730963 9568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "e0ce160a09b1462a91c99d4b674c713a"
format_stamp: "Formatted at 2024-05-03 08:13:50 on dist-test-slave-3g67"
server_key: "380495656f9f4e70db89037d299f1da5"
server_key_iv: "360a364013c1f7d7087cc703e7932029"
server_key_version: "encryptionkey@0"
I20240503 08:13:50.731036 9568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:50.757877 9568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:50.759940 9568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:50.760048 9568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:50.760254 9568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:50.760576 9568 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:50.760613 9568 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:50.760648 9568 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:50.760672 9568 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:50.766134 9568 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:34125
I20240503 08:13:50.766192 9696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:34125 every 8 connection(s)
I20240503 08:13:50.766510 9568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:13:50.768730 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 9568
I20240503 08:13:50.768817 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:13:50.768990 6419 external_mini_cluster.cc:1428] Setting key 122ebf4f45b5645af1a3295703b5378f
May 03 08:13:50 dist-test-slave-3g67 krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724030, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:50.776687 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:50.767318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:34847 (local address 127.6.68.254:43753)
0503 08:13:50.767399 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:50.767401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:50.767508 (+ 107us) server_negotiation.cc:407] Connection header received
0503 08:13:50.768041 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:50.768042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:50.768056 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:50.768105 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:50.769269 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:50.769936 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:50.770484 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:50.770687 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:50.772929 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:50.772934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:50.772934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:50.772944 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:50.774850 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:50.775514 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:50.775517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:50.775517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:50.775550 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:50.775843 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:50.775844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:50.775844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:50.776045 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:50.776143 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:50.776468 (+ 325us) server_negotiation.cc:299] Negotiation successful
0503 08:13:50.776570 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240503 08:13:50.776784 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:50.767253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43753 (local address 127.6.68.193:34847)
0503 08:13:50.767397 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:50.767424 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:50.767945 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:50.768158 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:50.768162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:50.768469 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:50.769153 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:50.769160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:50.770029 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:50.770030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:50.770407 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:50.770409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:50.770521 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:50.771081 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:50.771108 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:50.772821 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:50.774969 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:50.775006 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:50.775010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:50.775425 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:50.775647 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:50.775648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:50.775649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:50.775744 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:50.776178 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:50.776184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:50.776314 (+ 130us) client_negotiation.cc:769] Sending connection context
0503 08:13:50.776414 (+ 100us) client_negotiation.cc:240] Negotiation successful
0503 08:13:50.776573 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20240503 08:13:50.777328 9697 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43753
I20240503 08:13:50.777424 9697 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
I20240503 08:13:50.777659 9697 heartbeater.cc:511] Master 127.6.68.254:43753 requested a full tablet report, sending...
I20240503 08:13:50.778168 9505 ts_manager.cc:194] Registered new tserver with Master: e0ce160a09b1462a91c99d4b674c713a (127.6.68.193:34125)
I20240503 08:13:50.778851 9505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:34847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:13:50.798677 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43753
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:50.879052 9702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:50.879269 9702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:50.879293 9702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:50.880681 9702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:50.880738 9702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:50.880764 9702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:13:50.882316 9702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:50.882526 9702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:50.882732 9702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:50.887877 9707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:50.888460 9710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:50.889320 9708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:50.890096 9702 server_base.cc:1034] running on GCE node
I20240503 08:13:50.890338 9702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:50.890571 9702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:50.891705 9702 hybrid_clock.cc:648] HybridClock initialized: now 1714724030891690 us; error 43 us; skew 500 ppm
May 03 08:13:50 dist-test-slave-3g67 krb5kdc[9460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724030, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:50.894379 9702 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:13:50.894847 9702 webserver.cc:460] Webserver started at http://127.6.68.194:34965/ using document root <none> and password file <none>
I20240503 08:13:50.895035 9702 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:50.895089 9702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:50.895216 9702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:50.896301 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "487e60895a8642b496b82c4b1bea80dc"
format_stamp: "Formatted at 2024-05-03 08:13:50 on dist-test-slave-3g67"
server_key: "e21f01aea161413d5fb751f89d80a184"
server_key_iv: "0812ab4d846386d5ebbe2834153893f5"
server_key_version: "encryptionkey@0"
I20240503 08:13:50.896629 9702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "487e60895a8642b496b82c4b1bea80dc"
format_stamp: "Formatted at 2024-05-03 08:13:50 on dist-test-slave-3g67"
server_key: "e21f01aea161413d5fb751f89d80a184"
server_key_iv: "0812ab4d846386d5ebbe2834153893f5"
server_key_version: "encryptionkey@0"
I20240503 08:13:50.897902 9702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:13:50.898687 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:50.898887 9702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:50.898970 9702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "487e60895a8642b496b82c4b1bea80dc"
format_stamp: "Formatted at 2024-05-03 08:13:50 on dist-test-slave-3g67"
server_key: "e21f01aea161413d5fb751f89d80a184"
server_key_iv: "0812ab4d846386d5ebbe2834153893f5"
server_key_version: "encryptionkey@0"
I20240503 08:13:50.899036 9702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:50.916326 9702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:50.918681 9702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:50.918839 9702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:50.919052 9702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:50.919373 9702 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:50.919406 9702 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:50.919441 9702 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:50.919473 9702 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:50.925068 9702 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:44355
I20240503 08:13:50.925115 9830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:44355 every 8 connection(s)
I20240503 08:13:50.925429 9702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
May 03 08:13:50 dist-test-slave-3g67 krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724030, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:50.933699 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 9702
I20240503 08:13:50.933820 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:13:50.934051 6419 external_mini_cluster.cc:1428] Setting key c8352b848b4b6b17759d7bd2b7aa8bae
I20240503 08:13:50.934154 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:50.926152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:41315 (local address 127.6.68.254:43753)
0503 08:13:50.926233 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:50.926235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:50.926371 (+ 136us) server_negotiation.cc:407] Connection header received
0503 08:13:50.926881 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:50.926882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:50.926891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:50.926980 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:50.928164 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:50.928666 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:50.929219 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:50.929329 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:50.931417 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:50.931422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:50.931423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:50.931429 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:50.932882 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:50.933280 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:50.933281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:50.933281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:50.933308 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:50.933483 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:50.933484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:50.933484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:50.933583 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:50.933643 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:50.933951 (+ 308us) server_negotiation.cc:299] Negotiation successful
0503 08:13:50.934054 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20240503 08:13:50.934221 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:50.926089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43753 (local address 127.6.68.194:41315)
0503 08:13:50.926237 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:50.926276 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:50.926787 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:50.927018 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:50.927022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:50.927422 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:50.928049 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:50.928055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:50.928759 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:50.928759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:50.929141 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:50.929142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:50.929285 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:50.929817 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:50.929831 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:50.931291 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:50.932969 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:50.932977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:50.932980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:50.933208 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:50.933364 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:50.933365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:50.933366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:50.933428 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:50.933640 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:50.933645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:50.933772 (+ 127us) client_negotiation.cc:769] Sending connection context
0503 08:13:50.933883 (+ 111us) client_negotiation.cc:240] Negotiation successful
0503 08:13:50.934040 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20240503 08:13:50.934847 9831 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43753
I20240503 08:13:50.934952 9831 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:50.935148 9831 heartbeater.cc:511] Master 127.6.68.254:43753 requested a full tablet report, sending...
I20240503 08:13:50.935544 9505 ts_manager.cc:194] Registered new tserver with Master: 487e60895a8642b496b82c4b1bea80dc (127.6.68.194:44355)
I20240503 08:13:50.935953 9505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:41315
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:13:50.963299 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43753
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:51.043334 9836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:51.043548 9836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:51.043573 9836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:51.045022 9836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:51.045080 9836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:51.045107 9836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:13:51.046638 9836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:51.046851 9836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:51.047057 9836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:51.052173 9842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:51.052917 9844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:51.053572 9841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:51.054348 9836 server_base.cc:1034] running on GCE node
I20240503 08:13:51.054544 9836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:51.054751 9836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:51.055881 9836 hybrid_clock.cc:648] HybridClock initialized: now 1714724031055862 us; error 38 us; skew 500 ppm
May 03 08:13:51 dist-test-slave-3g67 krb5kdc[9460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724031, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:51.058609 9836 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:13:51.059028 9836 webserver.cc:460] Webserver started at http://127.6.68.195:39739/ using document root <none> and password file <none>
I20240503 08:13:51.059209 9836 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:51.059254 9836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:51.059367 9836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:51.060463 9836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "690f6529bf22438c8ac242cbb82505c8"
format_stamp: "Formatted at 2024-05-03 08:13:51 on dist-test-slave-3g67"
server_key: "c30da7181a7f142f179573fdd55ba06b"
server_key_iv: "577def79485120897bd3c5c42f0d4a90"
server_key_version: "encryptionkey@0"
I20240503 08:13:51.060793 9836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "690f6529bf22438c8ac242cbb82505c8"
format_stamp: "Formatted at 2024-05-03 08:13:51 on dist-test-slave-3g67"
server_key: "c30da7181a7f142f179573fdd55ba06b"
server_key_iv: "577def79485120897bd3c5c42f0d4a90"
server_key_version: "encryptionkey@0"
I20240503 08:13:51.062179 9836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:13:51.062975 9851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:51.063271 9836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:51.063372 9836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "690f6529bf22438c8ac242cbb82505c8"
format_stamp: "Formatted at 2024-05-03 08:13:51 on dist-test-slave-3g67"
server_key: "c30da7181a7f142f179573fdd55ba06b"
server_key_iv: "577def79485120897bd3c5c42f0d4a90"
server_key_version: "encryptionkey@0"
I20240503 08:13:51.063452 9836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:51.085381 9836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:51.087659 9836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:51.087815 9836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:51.088069 9836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:51.088414 9836 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:51.088454 9836 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:51.088490 9836 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:51.088521 9836 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:51.094627 9836 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:36735
I20240503 08:13:51.094684 9964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:36735 every 8 connection(s)
I20240503 08:13:51.095013 9836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:13:51.098030 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 9836
I20240503 08:13:51.098131 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:13:51.098275 6419 external_mini_cluster.cc:1428] Setting key e9278d3230553e053dbf59d7ff718a41
May 03 08:13:51 dist-test-slave-3g67 krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724031, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:51.103848 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:51.095848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:51429 (local address 127.6.68.254:43753)
0503 08:13:51.095918 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:51.095921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:51.096083 (+ 162us) server_negotiation.cc:407] Connection header received
0503 08:13:51.096638 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:51.096640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:51.096651 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:51.096715 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:51.097785 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:51.098423 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:51.099026 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:51.099203 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:51.101186 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:51.101190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:51.101191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:51.101197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:51.102696 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:51.103064 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:51.103065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:51.103065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:51.103083 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:51.103222 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:51.103222 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:51.103223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:51.103317 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:51.103364 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:51.103685 (+ 321us) server_negotiation.cc:299] Negotiation successful
0503 08:13:51.103778 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240503 08:13:51.103896 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:51.095805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43753 (local address 127.6.68.195:51429)
0503 08:13:51.095967 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:51.096002 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:51.096529 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:51.096785 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:51.096789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:51.097099 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:51.097674 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:51.097679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:51.098508 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:51.098509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:51.098954 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:51.098956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:51.099105 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:51.099651 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:51.099674 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:51.101096 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:51.102792 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:51.102798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:51.102801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:51.103004 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:51.103131 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:51.103132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:51.103132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:51.103183 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:51.103361 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:51.103364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:51.103474 (+ 110us) client_negotiation.cc:769] Sending connection context
0503 08:13:51.103560 (+ 86us) client_negotiation.cc:240] Negotiation successful
0503 08:13:51.103729 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20240503 08:13:51.104423 9965 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43753
I20240503 08:13:51.104509 9965 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:51.104727 9965 heartbeater.cc:511] Master 127.6.68.254:43753 requested a full tablet report, sending...
I20240503 08:13:51.105093 9505 ts_manager.cc:194] Registered new tserver with Master: 690f6529bf22438c8ac242cbb82505c8 (127.6.68.195:36735)
I20240503 08:13:51.105453 9505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:51429
I20240503 08:13:51.110260 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 03 08:13:51 dist-test-slave-3g67 krb5kdc[9460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724031, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
May 03 08:13:51 dist-test-slave-3g67 krb5kdc[9460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724031, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:51.132197 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:51.125471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.6.68.254:43753)
0503 08:13:51.125543 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:51.125546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:51.125616 (+ 70us) server_negotiation.cc:407] Connection header received
0503 08:13:51.125702 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:51.125703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:51.125715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:51.125767 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:51.126534 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:51.127022 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:51.127615 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:51.127795 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:51.129941 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:51.129946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:51.129947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:51.129953 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:51.131218 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:51.131506 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:51.131506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:51.131507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:51.131526 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:51.131708 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:51.131708 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:51.131709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:51.131869 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:51.131953 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:51.132055 (+ 102us) server_negotiation.cc:299] Negotiation successful
0503 08:13:51.132122 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20240503 08:13:51.132468 9505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41870
I20240503 08:13:51.132881 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 9568
I20240503 08:13:51.137712 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 9702
I20240503 08:13:51.142185 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 9836
I20240503 08:13:51.146539 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 9473
2024-05-03T08:13:51Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1838 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:13:51 dist-test-slave-3g67 krb5kdc[9977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:13:51 dist-test-slave-3g67 krb5kdc[9977](info): set up 2 sockets
May 03 08:13:51 dist-test-slave-3g67 krb5kdc[9977](info): commencing operation
krb5kdc: starting...
W20240503 08:13:52.107950 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:13:52.231884 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:44201
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:52.314210 9990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:52.314425 9990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:52.314448 9990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:52.315896 9990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:52.315950 9990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:52.315977 9990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:52.315997 9990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:52.316045 9990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:52.317755 9990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:52.318046 9990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:52.318246 9990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:52.323024 9995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:52.323474 9996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:52.324527 9998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:52.325255 9990 server_base.cc:1034] running on GCE node
I20240503 08:13:52.325510 9990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:52.325765 9990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:52.326891 9990 hybrid_clock.cc:648] HybridClock initialized: now 1714724032326876 us; error 34 us; skew 500 ppm
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:52.336871 9990 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:13:52.337342 9990 webserver.cc:460] Webserver started at http://127.6.68.254:35921/ using document root <none> and password file <none>
I20240503 08:13:52.337548 9990 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:52.337599 9990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:52.337716 9990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:52.338780 9990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "cab985ba48a04027b8de2ad8ddc92399"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "bb4d621825a772293094005a8f41d0e5"
server_key_iv: "1a5d4a18fdc480b5c8b9268dfade16b0"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.339100 9990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "cab985ba48a04027b8de2ad8ddc92399"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "bb4d621825a772293094005a8f41d0e5"
server_key_iv: "1a5d4a18fdc480b5c8b9268dfade16b0"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.340453 9990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:13:52.341235 10005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:52.341406 9990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:13:52.341488 9990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "cab985ba48a04027b8de2ad8ddc92399"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "bb4d621825a772293094005a8f41d0e5"
server_key_iv: "1a5d4a18fdc480b5c8b9268dfade16b0"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.341558 9990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:52.349959 9990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:52.352105 9990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:52.352244 9990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:52.356144 9990 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:44201
I20240503 08:13:52.356174 10057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:44201 every 8 connection(s)
I20240503 08:13:52.356513 9990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:13:52.356666 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 9990
I20240503 08:13:52.356767 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:13:52.356976 6419 external_mini_cluster.cc:1428] Setting key 916748320f8d58031abe2a70a56bfacf
I20240503 08:13:52.357244 10058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:52.360020 10058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399: Bootstrap starting.
I20240503 08:13:52.360774 10058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:52.361142 10058 log.cc:824] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:52.361961 10058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399: No bootstrap required, opened a new log
I20240503 08:13:52.364003 10058 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab985ba48a04027b8de2ad8ddc92399" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44201 } }
I20240503 08:13:52.364138 10058 raft_consensus.cc:374] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:52.364156 10058 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab985ba48a04027b8de2ad8ddc92399, State: Initialized, Role: FOLLOWER
I20240503 08:13:52.364253 10058 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab985ba48a04027b8de2ad8ddc92399" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44201 } }
I20240503 08:13:52.364318 10058 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:52.364346 10058 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:52.364389 10058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:52.364974 10058 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab985ba48a04027b8de2ad8ddc92399" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44201 } }
I20240503 08:13:52.365098 10058 leader_election.cc:302] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab985ba48a04027b8de2ad8ddc92399; no voters:
I20240503 08:13:52.365285 10058 leader_election.cc:288] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:52.365350 10063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [term 1 FOLLOWER]: Leader election won for term 1
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:52.365561 10058 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:52.365559 10063 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [term 1 LEADER]: Becoming Leader. State: Replica: cab985ba48a04027b8de2ad8ddc92399, State: Running, Role: LEADER
I20240503 08:13:52.365686 10063 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab985ba48a04027b8de2ad8ddc92399" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44201 } }
I20240503 08:13:52.366005 10064 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cab985ba48a04027b8de2ad8ddc92399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab985ba48a04027b8de2ad8ddc92399" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44201 } } }
I20240503 08:13:52.366004 10065 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab985ba48a04027b8de2ad8ddc92399. Latest consensus state: current_term: 1 leader_uuid: "cab985ba48a04027b8de2ad8ddc92399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab985ba48a04027b8de2ad8ddc92399" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44201 } } }
I20240503 08:13:52.366093 10064 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:52.366097 10065 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:52.367215 10069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:52.367686 10069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:52.369406 10069 catalog_manager.cc:1347] Generated new cluster ID: 0c15e03cd9c0488898d927def9cf949a
I20240503 08:13:52.369467 10069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:13:52.370705 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:52.357531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.6.68.254:44201)
0503 08:13:52.357713 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:52.357729 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:52.359895 (+ 2166us) server_negotiation.cc:407] Connection header received
0503 08:13:52.360532 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:52.360540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:52.360598 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:52.360774 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:52.361701 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.362860 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:52.363453 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.363804 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:52.366286 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:52.366299 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:52.366299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:52.366328 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:52.368820 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.369177 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.369180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.369181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.369229 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.369476 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.369477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.369478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.369738 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:52.369883 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:52.370393 (+ 510us) server_negotiation.cc:299] Negotiation successful
0503 08:13:52.370517 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20240503 08:13:52.385182 10069 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:52.386019 10069 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:52.393075 10069 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cab985ba48a04027b8de2ad8ddc92399: Generated new TSK 0
I20240503 08:13:52.393311 10069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:13:52.424229 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:44201
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:52.507265 10084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:52.507485 10084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:52.507522 10084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:52.508999 10084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:52.509061 10084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:52.509104 10084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:13:52.510809 10084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:52.511049 10084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:52.511260 10084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:52.516093 10092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:52.516872 10089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:52.517671 10090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:52.518523 10084 server_base.cc:1034] running on GCE node
I20240503 08:13:52.518779 10084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:52.519034 10084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:52.520179 10084 hybrid_clock.cc:648] HybridClock initialized: now 1714724032520167 us; error 36 us; skew 500 ppm
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:52.523260 10084 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:13:52.523847 10084 webserver.cc:460] Webserver started at http://127.6.68.193:42985/ using document root <none> and password file <none>
I20240503 08:13:52.524047 10084 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:52.524094 10084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:52.524211 10084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:52.525485 10084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "2220ab23582a408cb48899d8c7a939f7"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "ee082471f3f1849a509214961fbf3548"
server_key_iv: "aa9b7cf69af3b4cc3f56c9f7e9ce059c"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.525866 10084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "2220ab23582a408cb48899d8c7a939f7"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "ee082471f3f1849a509214961fbf3548"
server_key_iv: "aa9b7cf69af3b4cc3f56c9f7e9ce059c"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.527420 10084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:13:52.528183 10099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:52.528365 10084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:52.528443 10084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "2220ab23582a408cb48899d8c7a939f7"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "ee082471f3f1849a509214961fbf3548"
server_key_iv: "aa9b7cf69af3b4cc3f56c9f7e9ce059c"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.528517 10084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:52.547868 10084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:52.550115 10084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:52.550225 10084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:52.550457 10084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:52.550762 10084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:52.550794 10084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:52.550820 10084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:52.550858 10084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:52.556407 10084 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:38377
I20240503 08:13:52.556481 10212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:38377 every 8 connection(s)
I20240503 08:13:52.556767 10084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:13:52.559403 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 10084
I20240503 08:13:52.559520 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:13:52.559748 6419 external_mini_cluster.cc:1428] Setting key c4220e5bd9dbaeb07ab83ebc35951f62
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:52.566550 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:52.557572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:41865 (local address 127.6.68.254:44201)
0503 08:13:52.557639 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:52.557641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:52.557851 (+ 210us) server_negotiation.cc:407] Connection header received
0503 08:13:52.558292 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:52.558292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:52.558303 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:52.558371 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:52.559151 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.559713 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:52.560431 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.560642 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:52.563359 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:52.563364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:52.563365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:52.563374 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:52.565174 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.565612 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.565614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.565614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.565643 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.565835 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.565835 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.565836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.565968 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:52.566038 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:52.566309 (+ 271us) server_negotiation.cc:299] Negotiation successful
0503 08:13:52.566424 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240503 08:13:52.566581 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:52.557541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:44201 (local address 127.6.68.193:41865)
0503 08:13:52.557716 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:52.557748 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:52.558202 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:52.558428 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:52.558431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:52.558763 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:52.559070 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:52.559074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.559821 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:52.559823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:52.560323 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:52.560326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.560834 (+ 508us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:52.561473 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:52.561492 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:52.563236 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:52.565285 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:52.565292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:52.565295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:52.565533 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:52.565715 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:52.565715 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:52.565716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:52.565777 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:52.566034 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:52.566038 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:52.566157 (+ 119us) client_negotiation.cc:769] Sending connection context
0503 08:13:52.566268 (+ 111us) client_negotiation.cc:240] Negotiation successful
0503 08:13:52.566407 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20240503 08:13:52.567369 10213 heartbeater.cc:347] Connected to a master server at 127.6.68.254:44201
I20240503 08:13:52.567466 10213 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:52.567713 10213 heartbeater.cc:511] Master 127.6.68.254:44201 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
I20240503 08:13:52.568279 10022 ts_manager.cc:194] Registered new tserver with Master: 2220ab23582a408cb48899d8c7a939f7 (127.6.68.193:38377)
I20240503 08:13:52.568958 10022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:41865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:13:52.587800 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:44201
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:52.668867 10218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:52.669075 10218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:52.669109 10218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:52.670416 10218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:52.670467 10218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:52.670507 10218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:13:52.672184 10218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:52.672428 10218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:52.672621 10218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:52.677577 10223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:52.678016 10224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:52.678959 10226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:52.679751 10218 server_base.cc:1034] running on GCE node
I20240503 08:13:52.679946 10218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:52.680172 10218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:52.681304 10218 hybrid_clock.cc:648] HybridClock initialized: now 1714724032681287 us; error 36 us; skew 500 ppm
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:52.683807 10218 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:13:52.684258 10218 webserver.cc:460] Webserver started at http://127.6.68.194:33553/ using document root <none> and password file <none>
I20240503 08:13:52.684535 10218 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:52.684594 10218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:52.684705 10218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:52.686368 10218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "8d5e380cc7fd40ec90f19c8a71b1bcf3"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "0ce34c2f13bff228b8afc7e8d5029e93"
server_key_iv: "4ae95ee7e7d74a22f23eace7c5dc5d15"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.686707 10218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "8d5e380cc7fd40ec90f19c8a71b1bcf3"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "0ce34c2f13bff228b8afc7e8d5029e93"
server_key_iv: "4ae95ee7e7d74a22f23eace7c5dc5d15"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.687994 10218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:13:52.688704 10233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:52.688896 10218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:52.689028 10218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "8d5e380cc7fd40ec90f19c8a71b1bcf3"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "0ce34c2f13bff228b8afc7e8d5029e93"
server_key_iv: "4ae95ee7e7d74a22f23eace7c5dc5d15"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.689110 10218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:52.720294 10218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:52.722560 10218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:52.722698 10218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:52.722923 10218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:52.723224 10218 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:52.723258 10218 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:52.723335 10218 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:52.723356 10218 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:52.728874 10218 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:36927
I20240503 08:13:52.728925 10346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:36927 every 8 connection(s)
I20240503 08:13:52.729233 10218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:13:52.732796 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 10218
I20240503 08:13:52.732908 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:13:52.733098 6419 external_mini_cluster.cc:1428] Setting key 26c966053995d8029285edc2ff28b4b9
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:52.738241 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:52.729994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:59167 (local address 127.6.68.254:44201)
0503 08:13:52.730056 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:52.730058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:52.730209 (+ 151us) server_negotiation.cc:407] Connection header received
0503 08:13:52.730712 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:52.730714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:52.730724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:52.730790 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:52.731598 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.732153 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:52.732763 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.732956 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:52.735422 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:52.735426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:52.735427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:52.735435 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:52.737012 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.737399 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.737400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.737401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.737426 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.737601 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.737602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.737602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.737708 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:52.737768 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:52.738057 (+ 289us) server_negotiation.cc:299] Negotiation successful
0503 08:13:52.738147 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240503 08:13:52.738406 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:52.729951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:44201 (local address 127.6.68.194:59167)
0503 08:13:52.730098 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:52.730128 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:52.730581 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:52.730846 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:52.730850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:52.731148 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:52.731505 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:52.731510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.732257 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:52.732257 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:52.732663 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:52.732664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.733073 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:52.733668 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:52.733693 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:52.735302 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:52.737094 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:52.737101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:52.737105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:52.737328 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:52.737481 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:52.737482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:52.737483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:52.737545 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:52.737765 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:52.737771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:52.737929 (+ 158us) client_negotiation.cc:769] Sending connection context
0503 08:13:52.738032 (+ 103us) client_negotiation.cc:240] Negotiation successful
0503 08:13:52.738188 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20240503 08:13:52.739130 10347 heartbeater.cc:347] Connected to a master server at 127.6.68.254:44201
I20240503 08:13:52.739270 10347 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:52.739477 10347 heartbeater.cc:511] Master 127.6.68.254:44201 requested a full tablet report, sending...
I20240503 08:13:52.739926 10022 ts_manager.cc:194] Registered new tserver with Master: 8d5e380cc7fd40ec90f19c8a71b1bcf3 (127.6.68.194:36927)
I20240503 08:13:52.740535 10022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:59167
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:13:52.762569 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:44201
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:52.843981 10352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:52.844163 10352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:52.844188 10352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:52.845651 10352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:52.845701 10352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:52.845728 10352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:13:52.847329 10352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:52.847532 10352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:52.847707 10352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:52.852371 10357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:52.853004 10358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:52.853911 10360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:52.854801 10352 server_base.cc:1034] running on GCE node
I20240503 08:13:52.855062 10352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:52.855324 10352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:52.856501 10352 hybrid_clock.cc:648] HybridClock initialized: now 1714724032856481 us; error 34 us; skew 500 ppm
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:52.859315 10352 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:13:52.859819 10352 webserver.cc:460] Webserver started at http://127.6.68.195:40691/ using document root <none> and password file <none>
I20240503 08:13:52.860005 10352 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:52.860052 10352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:52.860178 10352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:52.861217 10352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "2c004f9a0e514be2bcf51aa2e811b222"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "3400db7da865eb31e44afe92a717b872"
server_key_iv: "5e1623d5650ad5ef53a2662c69a91b1f"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.861541 10352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "2c004f9a0e514be2bcf51aa2e811b222"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "3400db7da865eb31e44afe92a717b872"
server_key_iv: "5e1623d5650ad5ef53a2662c69a91b1f"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.862835 10352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:13:52.863574 10367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:52.863806 10352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:52.863884 10352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "2c004f9a0e514be2bcf51aa2e811b222"
format_stamp: "Formatted at 2024-05-03 08:13:52 on dist-test-slave-3g67"
server_key: "3400db7da865eb31e44afe92a717b872"
server_key_iv: "5e1623d5650ad5ef53a2662c69a91b1f"
server_key_version: "encryptionkey@0"
I20240503 08:13:52.863938 10352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:52.876608 10352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:52.878721 10352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:52.878827 10352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:52.879009 10352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:52.879323 10352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:52.879357 10352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:52.879395 10352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:52.879426 10352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:52.885102 10352 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:44699
I20240503 08:13:52.885172 10481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:44699 every 8 connection(s)
I20240503 08:13:52.885473 10352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:13:52.887329 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 10352
I20240503 08:13:52.887416 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:13:52.887589 6419 external_mini_cluster.cc:1428] Setting key 1e2af157824fc11bce60d4b88d3d9258
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:52.894260 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:52.886193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:60291 (local address 127.6.68.254:44201)
0503 08:13:52.886255 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:52.886257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:52.886448 (+ 191us) server_negotiation.cc:407] Connection header received
0503 08:13:52.886899 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:52.886900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:52.886908 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:52.886970 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:52.887985 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.888709 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:52.889250 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.889359 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:52.891617 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:52.891660 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:52.891661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:52.891682 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:52.893058 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.893464 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.893465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.893466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.893490 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.893695 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.893695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.893695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.893785 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:52.893835 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:52.894112 (+ 277us) server_negotiation.cc:299] Negotiation successful
0503 08:13:52.894192 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:13:52.894320 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:52.886166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:44201 (local address 127.6.68.195:60291)
0503 08:13:52.886341 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:52.886372 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:52.886818 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:52.887027 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:52.887031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:52.887400 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:52.887891 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:52.887896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.888797 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:52.888798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:52.889182 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:52.889183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.889540 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:52.890092 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:52.890105 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:52.891524 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:52.893174 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:52.893180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:52.893183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:52.893393 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:52.893553 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:52.893553 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:52.893554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:52.893633 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:52.893837 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:52.893840 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:52.893956 (+ 116us) client_negotiation.cc:769] Sending connection context
0503 08:13:52.894046 (+ 90us) client_negotiation.cc:240] Negotiation successful
0503 08:13:52.894164 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20240503 08:13:52.894748 10482 heartbeater.cc:347] Connected to a master server at 127.6.68.254:44201
I20240503 08:13:52.894850 10482 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:52.895061 10482 heartbeater.cc:511] Master 127.6.68.254:44201 requested a full tablet report, sending...
I20240503 08:13:52.895457 10022 ts_manager.cc:194] Registered new tserver with Master: 2c004f9a0e514be2bcf51aa2e811b222 (127.6.68.195:44699)
I20240503 08:13:52.895879 10022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:60291
I20240503 08:13:52.899600 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.193@KRBTEST.COM
I20240503 08:13:52.921594 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:52.913958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38184 (local address 127.6.68.193:38377)
0503 08:13:52.914115 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:52.914122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:52.914133 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:13:52.914194 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:52.914196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:52.914209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:52.914280 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:52.915018 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.915855 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:52.916459 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.916645 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:52.918783 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:52.918794 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:52.918794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:52.918801 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:52.920591 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.920951 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.920954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.920955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.920999 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.921126 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.921126 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.921127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.921262 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:52.921374 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:52.921431 (+ 57us) server_negotiation.cc:299] Negotiation successful
0503 08:13:52.921507 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
May 03 08:13:52 dist-test-slave-3g67 krb5kdc[9977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:52.929128 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:52.922861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.6.68.254:44201)
0503 08:13:52.922910 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:52.922912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:52.922921 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:13:52.922987 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:52.922988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:52.922996 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:52.923047 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:52.923606 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.924126 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:52.924804 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:52.924935 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:52.926903 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:52.926909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:52.926909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:52.926916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:52.928331 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.928622 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.928622 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.928622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.928642 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:52.928737 (+ 95us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:52.928737 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:52.928737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:52.928852 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:52.928931 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:52.928992 (+ 61us) server_negotiation.cc:299] Negotiation successful
0503 08:13:52.929052 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20240503 08:13:52.929414 10022 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35628
I20240503 08:13:52.929773 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 10084
I20240503 08:13:52.935560 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 10218
I20240503 08:13:52.940619 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 10352
I20240503 08:13:52.945381 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 9990
2024-05-03T08:13:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1798 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:13:53 dist-test-slave-3g67 krb5kdc[10494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:13:53 dist-test-slave-3g67 krb5kdc[10494](info): set up 2 sockets
May 03 08:13:53 dist-test-slave-3g67 krb5kdc[10494](info): commencing operation
krb5kdc: starting...
W20240503 08:13:53.908102 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:13:53 dist-test-slave-3g67 krb5kdc[10494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:13:54.035586 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40685
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:43901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:54.116901 10507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:54.117117 10507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:54.117142 10507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:54.118567 10507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:54.118628 10507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:54.118641 10507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:54.118654 10507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:54.118665 10507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:54.120461 10507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:54.120684 10507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:54.120905 10507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:54.125937 10515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:54.127101 10513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:54.127772 10512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:54.128499 10507 server_base.cc:1034] running on GCE node
I20240503 08:13:54.128803 10507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:54.129098 10507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:54.130303 10507 hybrid_clock.cc:648] HybridClock initialized: now 1714724034130280 us; error 45 us; skew 500 ppm
May 03 08:13:54 dist-test-slave-3g67 krb5kdc[10494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724034, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:54.133093 10507 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:13:54.133494 10507 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:54.133564 10507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:54.133671 10507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:54.134711 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "77c35df9bd95461a88f126427b6f3033"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "27eefac8bfcdcb171545594de1623879"
server_key_iv: "011b6bf4e665f046c7a4a7fb4d3467f3"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.135072 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "77c35df9bd95461a88f126427b6f3033"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "27eefac8bfcdcb171545594de1623879"
server_key_iv: "011b6bf4e665f046c7a4a7fb4d3467f3"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.136425 10507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:13:54.137185 10521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:54.137423 10507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:54.137513 10507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "77c35df9bd95461a88f126427b6f3033"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "27eefac8bfcdcb171545594de1623879"
server_key_iv: "011b6bf4e665f046c7a4a7fb4d3467f3"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.137590 10507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:54.156461 10507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:54.158670 10507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:54.158807 10507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:54.163002 10507 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:40685
I20240503 08:13:54.163029 10573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:40685 every 8 connection(s)
I20240503 08:13:54.163308 10507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:13:54.163936 10574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:54.166643 10574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033: Bootstrap starting.
I20240503 08:13:54.167199 10574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:54.167462 10574 log.cc:824] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:54.168190 10574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033: No bootstrap required, opened a new log
I20240503 08:13:54.169649 10574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c35df9bd95461a88f126427b6f3033" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40685 } }
I20240503 08:13:54.169780 10574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:54.169796 10574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c35df9bd95461a88f126427b6f3033, State: Initialized, Role: FOLLOWER
I20240503 08:13:54.169919 10574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c35df9bd95461a88f126427b6f3033" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40685 } }
I20240503 08:13:54.169989 10574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:54.170017 10574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:54.170044 10574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:54.170331 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 10507
I20240503 08:13:54.170454 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:13:54.170639 6419 external_mini_cluster.cc:1428] Setting key 0dc4d0e295e7e13d3f6f7367cb481253
I20240503 08:13:54.170894 10574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c35df9bd95461a88f126427b6f3033" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40685 } }
I20240503 08:13:54.171002 10574 leader_election.cc:302] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c35df9bd95461a88f126427b6f3033; no voters:
I20240503 08:13:54.171227 10574 leader_election.cc:288] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:54.171375 10578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:54.171514 10578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [term 1 LEADER]: Becoming Leader. State: Replica: 77c35df9bd95461a88f126427b6f3033, State: Running, Role: LEADER
I20240503 08:13:54.171584 10574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:54.171736 10578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c35df9bd95461a88f126427b6f3033" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40685 } }
I20240503 08:13:54.172282 10581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c35df9bd95461a88f126427b6f3033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c35df9bd95461a88f126427b6f3033" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40685 } } }
I20240503 08:13:54.172309 10580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c35df9bd95461a88f126427b6f3033. Latest consensus state: current_term: 1 leader_uuid: "77c35df9bd95461a88f126427b6f3033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c35df9bd95461a88f126427b6f3033" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40685 } } }
I20240503 08:13:54.172472 10581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:54.172483 10580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:54.172741 10586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:54.173264 10586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:54.175125 10586 catalog_manager.cc:1347] Generated new cluster ID: e6276ad96d7a4fca922528084c7340f8
I20240503 08:13:54.175232 10586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 03 08:13:54 dist-test-slave-3g67 krb5kdc[10494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:54.181560 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.171139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.6.68.254:40685)
0503 08:13:54.171356 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:54.171371 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:54.171399 (+ 28us) server_negotiation.cc:407] Connection header received
0503 08:13:54.171546 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:54.171553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:54.171608 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:54.172806 (+ 1198us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:54.172964 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.174169 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.174877 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.175129 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.177832 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:54.177861 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:54.177861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:54.177886 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:54.179959 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:54.180356 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:54.180358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:54.180359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:54.180413 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:54.180562 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:54.180562 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:54.180562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:54.180817 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:54.180966 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:54.181097 (+ 131us) server_negotiation.cc:299] Negotiation successful
0503 08:13:54.181226 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":99,"threads_started":1}
I20240503 08:13:54.217418 10586 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:54.218199 10586 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:54.226635 10586 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033: Generated new TSK 0
I20240503 08:13:54.226877 10586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:13:54.258762 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40685
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:43901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:54.345379 10600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:54.345599 10600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:54.345649 10600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:54.347020 10600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:54.347074 10600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:54.347118 10600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:13:54.348857 10600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:54.349114 10600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:54.349328 10600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:54.354440 10606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:54.355145 10605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:54.356093 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:54.356825 10600 server_base.cc:1034] running on GCE node
I20240503 08:13:54.357033 10600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:54.357280 10600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:54.358418 10600 hybrid_clock.cc:648] HybridClock initialized: now 1714724034358403 us; error 44 us; skew 500 ppm
May 03 08:13:54 dist-test-slave-3g67 krb5kdc[10494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724034, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:54.361230 10600 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:13:54.361644 10600 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:54.361698 10600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:54.361820 10600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:54.362855 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "068ab2e1c4fd4bd691597834ad4f4997"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "1eeabebdb1d935f46afcd93f98df67e5"
server_key_iv: "4cbe174dcca2eee463f09f7b0a62c812"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.363171 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "068ab2e1c4fd4bd691597834ad4f4997"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "1eeabebdb1d935f46afcd93f98df67e5"
server_key_iv: "4cbe174dcca2eee463f09f7b0a62c812"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.364456 10600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:13:54.365221 10614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:54.365386 10600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:54.365465 10600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "068ab2e1c4fd4bd691597834ad4f4997"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "1eeabebdb1d935f46afcd93f98df67e5"
server_key_iv: "4cbe174dcca2eee463f09f7b0a62c812"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.365528 10600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:54.382102 10600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:54.384497 10600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:54.384609 10600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:54.384809 10600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:54.385113 10600 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:54.385149 10600 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:54.385185 10600 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:54.385207 10600 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:54.390743 10600 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:42749
I20240503 08:13:54.390777 10727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:42749 every 8 connection(s)
I20240503 08:13:54.391202 10600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:13:54.393973 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 10600
I20240503 08:13:54.394073 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:13:54.394285 6419 external_mini_cluster.cc:1428] Setting key 34c094979bf31fde40d6f315b2f54dcf
May 03 08:13:54 dist-test-slave-3g67 krb5kdc[10494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724034, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:54.400065 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.392031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:59101 (local address 127.6.68.254:40685)
0503 08:13:54.392113 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:54.392116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:54.392224 (+ 108us) server_negotiation.cc:407] Connection header received
0503 08:13:54.392758 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:54.392759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:54.392773 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:54.392833 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:54.393627 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.394204 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.394907 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.395037 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.397266 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:54.397271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:54.397271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:54.397278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:54.398857 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:54.399237 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:54.399238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:54.399238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:54.399262 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:54.399436 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:54.399437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:54.399438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:54.399532 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:54.399592 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:54.399867 (+ 275us) server_negotiation.cc:299] Negotiation successful
0503 08:13:54.399970 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20240503 08:13:54.400274 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.391936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40685 (local address 127.6.68.193:59101)
0503 08:13:54.392104 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:54.392134 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:54.392618 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:54.392894 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:54.392899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:54.393236 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:54.393533 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.393537 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.394289 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.394290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:54.394809 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.394812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.395231 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.395773 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:54.395789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:54.397175 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:54.398948 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:54.398954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:54.398958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:54.399159 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:54.399320 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:54.399321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:54.399322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:54.399381 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:54.399588 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:54.399593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:54.399740 (+ 147us) client_negotiation.cc:769] Sending connection context
0503 08:13:54.399843 (+ 103us) client_negotiation.cc:240] Negotiation successful
0503 08:13:54.400010 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20240503 08:13:54.400722 10728 heartbeater.cc:347] Connected to a master server at 127.6.68.254:40685
I20240503 08:13:54.400815 10728 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:54.400997 10728 heartbeater.cc:511] Master 127.6.68.254:40685 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
I20240503 08:13:54.401497 10538 ts_manager.cc:194] Registered new tserver with Master: 068ab2e1c4fd4bd691597834ad4f4997 (127.6.68.193:42749)
I20240503 08:13:54.402154 10538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:59101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:13:54.422351 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40685
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:43901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:54.520731 10733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:54.520952 10733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:54.520978 10733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:54.522264 10733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:54.522305 10733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:54.522331 10733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:13:54.523944 10733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:54.524132 10733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:54.524286 10733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:54.529392 10741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:54.529987 10739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:54.530850 10738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:54.531543 10733 server_base.cc:1034] running on GCE node
I20240503 08:13:54.531775 10733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:54.531980 10733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:54.533130 10733 hybrid_clock.cc:648] HybridClock initialized: now 1714724034533098 us; error 46 us; skew 500 ppm
May 03 08:13:54 dist-test-slave-3g67 krb5kdc[10494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724034, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:54.535816 10733 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:13:54.536160 10733 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:54.536219 10733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:54.536343 10733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:54.537415 10733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "ba2d0c2c57da4e44a87c379298c3dd6e"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "f1b54dd267efb30c60c0527528a085c6"
server_key_iv: "cbc0c16112edd3b940215f7fe2d40207"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.537719 10733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "ba2d0c2c57da4e44a87c379298c3dd6e"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "f1b54dd267efb30c60c0527528a085c6"
server_key_iv: "cbc0c16112edd3b940215f7fe2d40207"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.538914 10733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240503 08:13:54.539659 10747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:54.539834 10733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:54.539911 10733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "ba2d0c2c57da4e44a87c379298c3dd6e"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "f1b54dd267efb30c60c0527528a085c6"
server_key_iv: "cbc0c16112edd3b940215f7fe2d40207"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.539990 10733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:54.565201 10733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:54.567163 10733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:54.567267 10733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:54.567512 10733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:54.567862 10733 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:54.567900 10733 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:54.567935 10733 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:54.567965 10733 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:54.573971 10733 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:44187
I20240503 08:13:54.574034 10860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:44187 every 8 connection(s)
I20240503 08:13:54.574268 10733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:13:54.577291 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 10733
I20240503 08:13:54.577379 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:13:54.577577 6419 external_mini_cluster.cc:1428] Setting key db9f67f84dc599264aea785f028aafec
May 03 08:13:54 dist-test-slave-3g67 krb5kdc[10494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724034, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:54.583217 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.574987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:36887 (local address 127.6.68.254:40685)
0503 08:13:54.575040 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:54.575044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:54.575222 (+ 178us) server_negotiation.cc:407] Connection header received
0503 08:13:54.575745 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:54.575746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:54.575759 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:54.575827 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:54.576607 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.577093 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.577688 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.577877 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.580434 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:54.580438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:54.580438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:54.580445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:54.581861 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:54.582382 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:54.582382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:54.582383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:54.582400 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:54.582610 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:54.582610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:54.582610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:54.582697 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:54.582740 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:54.583083 (+ 343us) server_negotiation.cc:299] Negotiation successful
0503 08:13:54.583149 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:13:54.583458 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.574949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40685 (local address 127.6.68.194:36887)
0503 08:13:54.575100 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:54.575132 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:54.575613 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:54.575906 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:54.575910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:54.576223 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:54.576508 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.576512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.577174 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.577175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:54.577588 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.577590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.578070 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.578674 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:54.578695 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:54.580312 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:54.581950 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:54.581958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:54.581963 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:54.582300 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:54.582457 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:54.582458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:54.582458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:54.582546 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:54.582751 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:54.582756 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:54.582933 (+ 177us) client_negotiation.cc:769] Sending connection context
0503 08:13:54.583070 (+ 137us) client_negotiation.cc:240] Negotiation successful
0503 08:13:54.583222 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20240503 08:13:54.583969 10861 heartbeater.cc:347] Connected to a master server at 127.6.68.254:40685
I20240503 08:13:54.584064 10861 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:54.584236 10861 heartbeater.cc:511] Master 127.6.68.254:40685 requested a full tablet report, sending...
I20240503 08:13:54.584645 10538 ts_manager.cc:194] Registered new tserver with Master: ba2d0c2c57da4e44a87c379298c3dd6e (127.6.68.194:44187)
I20240503 08:13:54.585029 10538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:36887
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:13:54.606709 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40685
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:43901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:13:54.687726 10866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:54.687930 10866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:54.687958 10866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:54.689267 10866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:54.689309 10866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:54.689335 10866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:13:54.690975 10866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:54.691151 10866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:54.691306 10866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:54.696291 10874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:54.696709 10872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:54.697470 10871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:54.698351 10866 server_base.cc:1034] running on GCE node
I20240503 08:13:54.698544 10866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:54.698786 10866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:54.699937 10866 hybrid_clock.cc:648] HybridClock initialized: now 1714724034699913 us; error 42 us; skew 500 ppm
May 03 08:13:54 dist-test-slave-3g67 krb5kdc[10494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724034, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:13:54.702500 10866 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:13:54.702780 10866 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:54.702818 10866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:54.702898 10866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:54.703944 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "2fa1c32859054b288545f4fa28223cd3"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "34d5942c31cfca119a6fd81a5003c8d0"
server_key_iv: "1e3557ff83e666c4aad45767a64fbb0e"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.704236 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "2fa1c32859054b288545f4fa28223cd3"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "34d5942c31cfca119a6fd81a5003c8d0"
server_key_iv: "1e3557ff83e666c4aad45767a64fbb0e"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.705401 10866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:54.706104 10880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:54.706315 10866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:54.706401 10866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "2fa1c32859054b288545f4fa28223cd3"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "34d5942c31cfca119a6fd81a5003c8d0"
server_key_iv: "1e3557ff83e666c4aad45767a64fbb0e"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.706475 10866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:54.730523 10866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:54.732650 10866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:54.732781 10866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:54.733028 10866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:54.733345 10866 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:54.733378 10866 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:54.733414 10866 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:54.733439 10866 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:54.739053 10866 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:40625
I20240503 08:13:54.739137 10993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:40625 every 8 connection(s)
I20240503 08:13:54.739362 10866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:13:54.741562 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 10866
I20240503 08:13:54.741709 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:13:54.741885 6419 external_mini_cluster.cc:1428] Setting key 1effbe061be5e03bb045f2307a29e2fa
May 03 08:13:54 dist-test-slave-3g67 krb5kdc[10494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724034, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:13:54.748178 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.740090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:46355 (local address 127.6.68.254:40685)
0503 08:13:54.740158 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:54.740161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:54.740416 (+ 255us) server_negotiation.cc:407] Connection header received
0503 08:13:54.740939 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:54.740940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:54.740952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:54.741014 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:54.741880 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.742399 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.743005 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.743191 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.745557 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:54.745561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:54.745561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:54.745567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:54.746956 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:54.747292 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:54.747293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:54.747293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:54.747310 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:54.747493 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:54.747494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:54.747495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:54.747597 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:54.747740 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:54.748018 (+ 278us) server_negotiation.cc:299] Negotiation successful
0503 08:13:54.748102 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:13:54.748214 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.740042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40685 (local address 127.6.68.195:46355)
0503 08:13:54.740241 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:54.740276 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:54.740817 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:54.741080 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:54.741084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:54.741447 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:54.741776 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.741781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.742486 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.742487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:54.742920 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.742921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.743359 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.743997 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:13:54.744018 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:54.745465 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:54.747034 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:54.747039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:54.747042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:54.747232 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:54.747367 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:13:54.747367 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:13:54.747368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:13:54.747418 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:13:54.747744 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:54.747747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:54.747865 (+ 118us) client_negotiation.cc:769] Sending connection context
0503 08:13:54.747957 (+ 92us) client_negotiation.cc:240] Negotiation successful
0503 08:13:54.748069 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20240503 08:13:54.748757 10994 heartbeater.cc:347] Connected to a master server at 127.6.68.254:40685
I20240503 08:13:54.748864 10994 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:54.749043 10994 heartbeater.cc:511] Master 127.6.68.254:40685 requested a full tablet report, sending...
I20240503 08:13:54.749454 10538 ts_manager.cc:194] Registered new tserver with Master: 2fa1c32859054b288545f4fa28223cd3 (127.6.68.195:40625)
I20240503 08:13:54.749850 10538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:46355
I20240503 08:13:54.753675 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:13:54.760541 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.754996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.6.68.254:40685)
0503 08:13:54.755044 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:54.755045 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:54.755124 (+ 79us) server_negotiation.cc:407] Connection header received
0503 08:13:54.755210 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:54.755211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:54.755220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:54.755267 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:54.756060 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.756528 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.757087 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.757185 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.758248 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:54.758251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:54.758252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:13:54.758257 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:54.759642 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:54.759973 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:54.759974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:54.759975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:54.759996 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:13:54.760112 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:13:54.760113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:13:54.760113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:13:54.760242 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:54.760323 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:54.760401 (+ 78us) server_negotiation.cc:299] Negotiation successful
0503 08:13:54.760468 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240503 08:13:54.762434 10538 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:13:54.763944 10538 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:13:54.769598 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.766240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:44187 (local address 127.0.0.1:57678)
0503 08:13:54.766363 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:54.766391 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:54.766474 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:54.766921 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:54.766924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:54.766926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:54.767178 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.767180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.768280 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.768281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:54.769124 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.769126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.769341 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.769385 (+ 44us) client_negotiation.cc:769] Sending connection context
0503 08:13:54.769445 (+ 60us) client_negotiation.cc:240] Negotiation successful
0503 08:13:54.769485 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20240503 08:13:54.769908 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.766240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:42749 (local address 127.0.0.1:35642)
0503 08:13:54.766559 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:54.766570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:54.766620 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:54.766923 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:54.766924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:54.766930 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:54.767256 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.767259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.768704 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.768705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:54.769696 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.769698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.769761 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.769765 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:13:54.769797 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:13:54.769831 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":4,"thread_start_us":115,"threads_started":1}
I20240503 08:13:54.770139 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.766431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.6.68.194:44187)
0503 08:13:54.766745 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:54.766752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:54.766762 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:13:54.766795 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:54.766796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:54.766811 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:54.767264 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:54.767339 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.768202 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.769402 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.769929 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.769986 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:54.770019 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:13:54.770062 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":90,"threads_started":1}
I20240503 08:13:54.770434 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.766443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:40625 (local address 127.0.0.1:54588)
0503 08:13:54.766573 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:54.766582 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:54.766622 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:54.767046 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:54.767047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:54.767049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:54.767264 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.767266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.768700 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.768701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:54.769392 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.769394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.769463 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.769466 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:13:54.769498 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:13:54.769538 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20240503 08:13:54.770819 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.766338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.6.68.193:42749)
0503 08:13:54.766731 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:54.766740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:54.766751 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:13:54.766790 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:54.766791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:54.766808 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:54.766904 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:54.767368 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.768573 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.770126 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.770671 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.770707 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:54.770733 (+ 26us) server_negotiation.cc:299] Negotiation successful
0503 08:13:54.770763 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":111,"threads_started":1}
I20240503 08:13:54.770767 10795 tablet_service.cc:1467] Processing CreateTablet for tablet 5d57d05055584ed194169e56bc482a9a (DEFAULT_TABLE table=test-table [id=2faac47b50e142ea96d9e3e8f651cb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:54.771095 10795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d57d05055584ed194169e56bc482a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:54.771124 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.766640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.6.68.195:40625)
0503 08:13:54.766787 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:54.766794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:54.766802 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:13:54.766844 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:54.766845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:54.766857 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:54.766967 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:54.767354 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.768599 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.769968 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.770887 (+ 919us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.770929 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:54.770962 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:13:54.771005 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20240503 08:13:54.771301 10662 tablet_service.cc:1467] Processing CreateTablet for tablet 5d57d05055584ed194169e56bc482a9a (DEFAULT_TABLE table=test-table [id=2faac47b50e142ea96d9e3e8f651cb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:54.771571 10662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d57d05055584ed194169e56bc482a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:54.771603 10928 tablet_service.cc:1467] Processing CreateTablet for tablet 5d57d05055584ed194169e56bc482a9a (DEFAULT_TABLE table=test-table [id=2faac47b50e142ea96d9e3e8f651cb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:54.771888 10928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d57d05055584ed194169e56bc482a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:54.774077 11011 tablet_bootstrap.cc:492] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997: Bootstrap starting.
I20240503 08:13:54.774453 11012 tablet_bootstrap.cc:492] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: Bootstrap starting.
I20240503 08:13:54.774760 11011 tablet_bootstrap.cc:654] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:54.775007 11012 tablet_bootstrap.cc:654] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:54.775252 11011 log.cc:824] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:54.775379 11012 log.cc:824] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:54.775722 11013 tablet_bootstrap.cc:492] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3: Bootstrap starting.
I20240503 08:13:54.776062 11011 tablet_bootstrap.cc:492] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997: No bootstrap required, opened a new log
I20240503 08:13:54.776083 11012 tablet_bootstrap.cc:492] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: No bootstrap required, opened a new log
I20240503 08:13:54.776134 11011 ts_tablet_manager.cc:1367] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:54.776134 11012 ts_tablet_manager.cc:1367] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:54.776208 11013 tablet_bootstrap.cc:654] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:54.776412 11013 log.cc:824] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:54.776988 11013 tablet_bootstrap.cc:492] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3: No bootstrap required, opened a new log
I20240503 08:13:54.777029 11013 ts_tablet_manager.cc:1367] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:13:54.778563 11011 raft_consensus.cc:348] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068ab2e1c4fd4bd691597834ad4f4997" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42749 } } peers { permanent_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 44187 } } peers { permanent_uuid: "2fa1c32859054b288545f4fa28223cd3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 40625 } }
I20240503 08:13:54.778563 11012 raft_consensus.cc:348] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068ab2e1c4fd4bd691597834ad4f4997" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42749 } } peers { permanent_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 44187 } } peers { permanent_uuid: "2fa1c32859054b288545f4fa28223cd3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 40625 } }
I20240503 08:13:54.778702 11012 raft_consensus.cc:374] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:54.778702 11011 raft_consensus.cc:374] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:54.778731 11011 raft_consensus.cc:729] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068ab2e1c4fd4bd691597834ad4f4997, State: Initialized, Role: FOLLOWER
I20240503 08:13:54.778731 11012 raft_consensus.cc:729] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba2d0c2c57da4e44a87c379298c3dd6e, State: Initialized, Role: FOLLOWER
I20240503 08:13:54.778693 11013 raft_consensus.cc:348] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068ab2e1c4fd4bd691597834ad4f4997" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42749 } } peers { permanent_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 44187 } } peers { permanent_uuid: "2fa1c32859054b288545f4fa28223cd3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 40625 } }
I20240503 08:13:54.778823 11013 raft_consensus.cc:374] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:54.778854 11013 raft_consensus.cc:729] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa1c32859054b288545f4fa28223cd3, State: Initialized, Role: FOLLOWER
I20240503 08:13:54.778836 11011 consensus_queue.cc:260] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068ab2e1c4fd4bd691597834ad4f4997" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42749 } } peers { permanent_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 44187 } } peers { permanent_uuid: "2fa1c32859054b288545f4fa28223cd3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 40625 } }
I20240503 08:13:54.778833 11012 consensus_queue.cc:260] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068ab2e1c4fd4bd691597834ad4f4997" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42749 } } peers { permanent_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 44187 } } peers { permanent_uuid: "2fa1c32859054b288545f4fa28223cd3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 40625 } }
I20240503 08:13:54.778949 11013 consensus_queue.cc:260] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068ab2e1c4fd4bd691597834ad4f4997" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42749 } } peers { permanent_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 44187 } } peers { permanent_uuid: "2fa1c32859054b288545f4fa28223cd3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 40625 } }
I20240503 08:13:54.779093 11011 ts_tablet_manager.cc:1398] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:54.779150 10728 heartbeater.cc:503] Master 127.6.68.254:40685 was elected leader, sending a full tablet report...
I20240503 08:13:54.779188 11013 ts_tablet_manager.cc:1398] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:54.779305 10994 heartbeater.cc:503] Master 127.6.68.254:40685 was elected leader, sending a full tablet report...
I20240503 08:13:54.779383 10861 heartbeater.cc:503] Master 127.6.68.254:40685 was elected leader, sending a full tablet report...
I20240503 08:13:54.779093 11012 ts_tablet_manager.cc:1398] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:54.820858 11017 raft_consensus.cc:482] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:54.821002 11017 raft_consensus.cc:504] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068ab2e1c4fd4bd691597834ad4f4997" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42749 } } peers { permanent_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 44187 } } peers { permanent_uuid: "2fa1c32859054b288545f4fa28223cd3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 40625 } }
I20240503 08:13:54.821367 11017 leader_election.cc:288] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 068ab2e1c4fd4bd691597834ad4f4997 (127.6.68.193:42749), 2fa1c32859054b288545f4fa28223cd3 (127.6.68.195:40625)
I20240503 08:13:54.823694 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.821573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:42749 (local address 127.6.68.194:51407)
0503 08:13:54.821636 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:54.821648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:54.821696 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:54.821862 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:54.821863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:54.821865 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:54.822105 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.822107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.822854 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.822855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:54.823482 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.823484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.823548 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.823552 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:13:54.823570 (+ 18us) client_negotiation.cc:240] Negotiation successful
0503 08:13:54.823596 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20240503 08:13:54.823809 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.821573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:40625 (local address 127.6.68.194:52753)
0503 08:13:54.821814 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:54.821824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:54.821891 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:54.822018 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:54.822018 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:54.822019 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:13:54.822303 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.822306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.822966 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.822967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:54.823568 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:54.823570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.823663 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.823666 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:13:54.823699 (+ 33us) client_negotiation.cc:240] Negotiation successful
0503 08:13:54.823734 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":15,"thread_start_us":133,"threads_started":1}
I20240503 08:13:54.824191 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.821646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:51407 (local address 127.6.68.193:42749)
0503 08:13:54.821701 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:54.821703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:54.821719 (+ 16us) server_negotiation.cc:407] Connection header received
0503 08:13:54.821755 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:54.821756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:54.821766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:54.821846 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:54.822183 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.822756 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.823597 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.824091 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.824099 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:54.824120 (+ 21us) server_negotiation.cc:299] Negotiation successful
0503 08:13:54.824141 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240503 08:13:54.824275 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.821646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:52753 (local address 127.6.68.195:40625)
0503 08:13:54.821726 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:54.821728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:54.821907 (+ 179us) server_negotiation.cc:407] Connection header received
0503 08:13:54.821942 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:54.821943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:54.821953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:54.822012 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:13:54.822382 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.822874 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.823666 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.824160 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.824167 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:54.824186 (+ 19us) server_negotiation.cc:299] Negotiation successful
0503 08:13:54.824209 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20240503 08:13:54.824497 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d57d05055584ed194169e56bc482a9a" candidate_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068ab2e1c4fd4bd691597834ad4f4997" is_pre_election: true
I20240503 08:13:54.824564 10948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d57d05055584ed194169e56bc482a9a" candidate_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa1c32859054b288545f4fa28223cd3" is_pre_election: true
I20240503 08:13:54.824642 10682 raft_consensus.cc:2454] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba2d0c2c57da4e44a87c379298c3dd6e in term 0.
I20240503 08:13:54.824682 10948 raft_consensus.cc:2454] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba2d0c2c57da4e44a87c379298c3dd6e in term 0.
W20240503 08:13:54.824837 10862 tablet.cc:2369] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:13:54.824906 10750 leader_election.cc:302] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068ab2e1c4fd4bd691597834ad4f4997, ba2d0c2c57da4e44a87c379298c3dd6e; no voters:
I20240503 08:13:54.825028 11017 raft_consensus.cc:2781] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:54.825088 11017 raft_consensus.cc:482] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:54.825121 11017 raft_consensus.cc:3037] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:54.825848 11017 raft_consensus.cc:504] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068ab2e1c4fd4bd691597834ad4f4997" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42749 } } peers { permanent_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 44187 } } peers { permanent_uuid: "2fa1c32859054b288545f4fa28223cd3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 40625 } }
I20240503 08:13:54.825974 11017 leader_election.cc:288] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [CANDIDATE]: Term 1 election: Requested vote from peers 068ab2e1c4fd4bd691597834ad4f4997 (127.6.68.193:42749), 2fa1c32859054b288545f4fa28223cd3 (127.6.68.195:40625)
I20240503 08:13:54.826136 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d57d05055584ed194169e56bc482a9a" candidate_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068ab2e1c4fd4bd691597834ad4f4997"
I20240503 08:13:54.826148 10948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d57d05055584ed194169e56bc482a9a" candidate_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa1c32859054b288545f4fa28223cd3"
I20240503 08:13:54.826223 10948 raft_consensus.cc:3037] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:54.826224 10682 raft_consensus.cc:3037] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:54.826903 10948 raft_consensus.cc:2454] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba2d0c2c57da4e44a87c379298c3dd6e in term 1.
I20240503 08:13:54.826997 10682 raft_consensus.cc:2454] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba2d0c2c57da4e44a87c379298c3dd6e in term 1.
I20240503 08:13:54.827090 10748 leader_election.cc:302] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa1c32859054b288545f4fa28223cd3, ba2d0c2c57da4e44a87c379298c3dd6e; no voters:
I20240503 08:13:54.827201 11017 raft_consensus.cc:2781] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:54.827517 11017 raft_consensus.cc:686] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 1 LEADER]: Becoming Leader. State: Replica: ba2d0c2c57da4e44a87c379298c3dd6e, State: Running, Role: LEADER
I20240503 08:13:54.827669 11017 consensus_queue.cc:237] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068ab2e1c4fd4bd691597834ad4f4997" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42749 } } peers { permanent_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 44187 } } peers { permanent_uuid: "2fa1c32859054b288545f4fa28223cd3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 40625 } }
I20240503 08:13:54.828483 10538 catalog_manager.cc:5484] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e reported cstate change: term changed from 0 to 1, leader changed from <none> to ba2d0c2c57da4e44a87c379298c3dd6e (127.6.68.194). New cstate: current_term: 1 leader_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "068ab2e1c4fd4bd691597834ad4f4997" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba2d0c2c57da4e44a87c379298c3dd6e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa1c32859054b288545f4fa28223cd3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 40625 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:54.845568 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.843499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.6.68.194:44187)
0503 08:13:54.843558 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:54.843561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:54.843570 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:13:54.843614 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:54.843614 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:54.843684 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:54.843750 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:13:54.843991 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.844487 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:54.845064 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:54.845205 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:54.845244 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:13:54.845334 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:13:54.845401 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:54.845487 (+ 86us) server_negotiation.cc:299] Negotiation successful
0503 08:13:54.845505 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240503 08:13:54.847299 10682 raft_consensus.cc:1261] T 5d57d05055584ed194169e56bc482a9a P 068ab2e1c4fd4bd691597834ad4f4997 [term 1 FOLLOWER]: Refusing update from remote peer ba2d0c2c57da4e44a87c379298c3dd6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:54.847299 10948 raft_consensus.cc:1261] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3 [term 1 FOLLOWER]: Refusing update from remote peer ba2d0c2c57da4e44a87c379298c3dd6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:54.847663 11021 consensus_queue.cc:1035] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa1c32859054b288545f4fa28223cd3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 40625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:54.847849 11017 consensus_queue.cc:1035] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "068ab2e1c4fd4bd691597834ad4f4997" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:54.850139 11024 mvcc.cc:205] Tried to move back new op lower bound from 7023509646731988992 to 7023509646654734336. Current Snapshot: MvccSnapshot[applied={T|T < 7023509646731988992}]
I20240503 08:13:54.850379 11025 mvcc.cc:205] Tried to move back new op lower bound from 7023509646731988992 to 7023509646654734336. Current Snapshot: MvccSnapshot[applied={T|T < 7023509646731988992}]
I20240503 08:13:54.853112 10536 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49332:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:54.853209 10536 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49332:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:54.854000 10536 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 77c35df9bd95461a88f126427b6f3033: Sending DeleteTablet for 3 replicas of tablet 5d57d05055584ed194169e56bc482a9a
I20240503 08:13:54.854396 10662 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d57d05055584ed194169e56bc482a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:54 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:35642
I20240503 08:13:54.854404 10795 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d57d05055584ed194169e56bc482a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:54 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:57678
I20240503 08:13:54.854466 10928 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d57d05055584ed194169e56bc482a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:54 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:54588
I20240503 08:13:54.854677 11034 tablet_replica.cc:330] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: stopping tablet replica
I20240503 08:13:54.854866 11035 tablet_replica.cc:330] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3: stopping tablet replica
I20240503 08:13:54.854951 11035 raft_consensus.cc:2229] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:54.854979 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 10600
I20240503 08:13:54.855046 11035 raft_consensus.cc:2258] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:54.856156 11035 ts_tablet_manager.cc:1875] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:54.858122 11035 ts_tablet_manager.cc:1888] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:13:54.858191 11035 log.cc:1196] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/5d57d05055584ed194169e56bc482a9a
I20240503 08:13:54.858465 11035 ts_tablet_manager.cc:1909] T 5d57d05055584ed194169e56bc482a9a P 2fa1c32859054b288545f4fa28223cd3: Deleting consensus metadata
I20240503 08:13:54.859727 11034 raft_consensus.cc:2229] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:54.859915 11034 raft_consensus.cc:2258] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:54.860299 10522 catalog_manager.cc:4830] TS 2fa1c32859054b288545f4fa28223cd3 (127.6.68.195:40625): tablet 5d57d05055584ed194169e56bc482a9a (table test-table [id=2faac47b50e142ea96d9e3e8f651cb60]) successfully deleted
I20240503 08:13:54.860574 11034 ts_tablet_manager.cc:1875] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:54.862615 11034 ts_tablet_manager.cc:1888] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:13:54.862708 11034 log.cc:1196] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/5d57d05055584ed194169e56bc482a9a
W20240503 08:13:54.862900 10524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:42749 (error 108)
I20240503 08:13:54.863013 11034 ts_tablet_manager.cc:1909] T 5d57d05055584ed194169e56bc482a9a P ba2d0c2c57da4e44a87c379298c3dd6e: Deleting consensus metadata
I20240503 08:13:54.863447 10522 catalog_manager.cc:4830] TS ba2d0c2c57da4e44a87c379298c3dd6e (127.6.68.194:44187): tablet 5d57d05055584ed194169e56bc482a9a (table test-table [id=2faac47b50e142ea96d9e3e8f651cb60]) successfully deleted
I20240503 08:13:54.863567 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 10733
I20240503 08:13:54.863680 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:54.863529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:42749 (local address 127.0.0.1:35648)
0503 08:13:54.863583 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:54.863614 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20240503 08:13:54.863778 10524 catalog_manager.cc:4573] TS 068ab2e1c4fd4bd691597834ad4f4997 (127.6.68.193:42749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d57d05055584ed194169e56bc482a9a: Network error: Client connection negotiation failed: client connection to 127.6.68.193:42749: connect: Connection refused (error 111)
I20240503 08:13:54.867990 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 10866
I20240503 08:13:54.872700 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 10507
2024-05-03T08:13:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1927 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2024-05-03T08:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:54Z Disabled control of system clock
I20240503 08:13:54.899538 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:36317
--webserver_interface=127.6.68.254
--webserver_port=0
--builtin_ntp_servers=127.6.68.212:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:36317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240503 08:13:54.980152 11040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:54.980351 11040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:54.980373 11040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:54.981715 11040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:54.981763 11040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:54.981776 11040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:54.981788 11040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:54.981799 11040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:54.983310 11040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:36317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:54.983525 11040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:54.983768 11040 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:54.988294 11048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:54.988852 11045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:54.989459 11046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:54.990479 11040 server_base.cc:1034] running on GCE node
I20240503 08:13:54.990762 11040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:54.991022 11040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:54.992197 11040 hybrid_clock.cc:648] HybridClock initialized: now 1714724034992151 us; error 44 us; skew 500 ppm
I20240503 08:13:54.993700 11040 webserver.cc:460] Webserver started at http://127.6.68.254:44733/ using document root <none> and password file <none>
I20240503 08:13:54.993904 11040 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:54.993947 11040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:54.994062 11040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:54.995153 11040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "c0d91df0a8b64878a700e348a0790927"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "e05400bbe7e14c95ac072aa61a2cee50"
server_key_iv: "b6e9bd571854f4db884dcfda0301a4ac"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.995481 11040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "c0d91df0a8b64878a700e348a0790927"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "e05400bbe7e14c95ac072aa61a2cee50"
server_key_iv: "b6e9bd571854f4db884dcfda0301a4ac"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.996791 11040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:54.997594 11054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:54.997783 11040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:54.997864 11040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "c0d91df0a8b64878a700e348a0790927"
format_stamp: "Formatted at 2024-05-03 08:13:54 on dist-test-slave-3g67"
server_key: "e05400bbe7e14c95ac072aa61a2cee50"
server_key_iv: "b6e9bd571854f4db884dcfda0301a4ac"
server_key_version: "encryptionkey@0"
I20240503 08:13:54.997946 11040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:55.001793 11040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:55.002076 11040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:55.002188 11040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:55.005903 11040 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:36317
I20240503 08:13:55.005931 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:36317 every 8 connection(s)
I20240503 08:13:55.006259 11040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:13:55.006857 11107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:55.009327 11107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927: Bootstrap starting.
I20240503 08:13:55.009856 11107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:55.010107 11107 log.cc:824] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:55.010768 11107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927: No bootstrap required, opened a new log
I20240503 08:13:55.012276 11107 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d91df0a8b64878a700e348a0790927" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36317 } }
I20240503 08:13:55.012408 11107 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:55.012424 11107 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d91df0a8b64878a700e348a0790927, State: Initialized, Role: FOLLOWER
I20240503 08:13:55.012547 11107 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d91df0a8b64878a700e348a0790927" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36317 } }
I20240503 08:13:55.012609 11107 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:55.012629 11107 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:55.012679 11107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:55.013231 11107 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d91df0a8b64878a700e348a0790927" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36317 } }
I20240503 08:13:55.013319 11107 leader_election.cc:302] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d91df0a8b64878a700e348a0790927; no voters:
I20240503 08:13:55.013507 11107 leader_election.cc:288] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:55.013618 11110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:55.013787 11110 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [term 1 LEADER]: Becoming Leader. State: Replica: c0d91df0a8b64878a700e348a0790927, State: Running, Role: LEADER
I20240503 08:13:55.013787 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 11040
I20240503 08:13:55.013888 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:13:55.013857 11107 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:55.014075 6419 external_mini_cluster.cc:1428] Setting key ca7e2a91cdcb66bf862d008c3006c47a
I20240503 08:13:55.014601 11110 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d91df0a8b64878a700e348a0790927" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36317 } }
I20240503 08:13:55.015275 11112 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d91df0a8b64878a700e348a0790927. Latest consensus state: current_term: 1 leader_uuid: "c0d91df0a8b64878a700e348a0790927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d91df0a8b64878a700e348a0790927" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36317 } } }
I20240503 08:13:55.015332 11111 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0d91df0a8b64878a700e348a0790927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d91df0a8b64878a700e348a0790927" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36317 } } }
I20240503 08:13:55.015482 11112 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:55.015492 11111 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927 [sys.catalog]: This master's current role is: LEADER
W20240503 08:13:55.016269 11126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:13:55.016342 11126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:13:55.016448 11127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:55.016611 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.014577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.6.68.254:36317)
0503 08:13:55.015050 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.015058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.015080 (+ 22us) server_negotiation.cc:407] Connection header received
0503 08:13:55.015199 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.015207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.015250 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.015421 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.016202 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.016205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.016205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.016209 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.016240 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.016247 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.016320 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.016444 (+ 124us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.016491 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":70,"threads_started":1}
I20240503 08:13:55.016902 11127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:55.018347 11127 catalog_manager.cc:1347] Generated new cluster ID: 4c71d5e0458d45dba4f5c6fb48e89ae2
I20240503 08:13:55.018400 11127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:13:55.037508 11127 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:55.037681 11127 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:55.043114 11127 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927: Generated new TSK 0
I20240503 08:13:55.043326 11127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:13:55.053910 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36317
--builtin_ntp_servers=127.6.68.212:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240503 08:13:55.147454 11131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:55.147675 11131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:55.147715 11131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:55.149092 11131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:55.149142 11131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:55.149168 11131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:13:55.150647 11131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:55.150876 11131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:55.151132 11131 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:55.155990 11139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.156672 11136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.157578 11137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:55.158258 11131 server_base.cc:1034] running on GCE node
I20240503 08:13:55.158437 11131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:55.158656 11131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:55.159794 11131 hybrid_clock.cc:648] HybridClock initialized: now 1714724035159782 us; error 38 us; skew 500 ppm
I20240503 08:13:55.161108 11131 webserver.cc:460] Webserver started at http://127.6.68.193:40843/ using document root <none> and password file <none>
I20240503 08:13:55.161321 11131 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:55.161362 11131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:55.161458 11131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:55.162456 11131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "81f97821481343a3b487dbb08707ad20"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "ee61f0e33052de14af744970ae6e217c"
server_key_iv: "0c294c69666d030cef0b7ed60770b349"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.162760 11131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "81f97821481343a3b487dbb08707ad20"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "ee61f0e33052de14af744970ae6e217c"
server_key_iv: "0c294c69666d030cef0b7ed60770b349"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.163997 11131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:13:55.164685 11145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.164858 11131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:13:55.164930 11131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "81f97821481343a3b487dbb08707ad20"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "ee61f0e33052de14af744970ae6e217c"
server_key_iv: "0c294c69666d030cef0b7ed60770b349"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.164997 11131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:55.168503 11131 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:55.168730 11131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:55.168824 11131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:55.169044 11131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:55.169365 11131 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:55.169401 11131 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.169432 11131 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:55.169461 11131 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.176076 11131 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:39833
I20240503 08:13:55.176100 11258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:39833 every 8 connection(s)
I20240503 08:13:55.176486 11131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:13:55.179152 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.177212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:52937 (local address 127.6.68.254:36317)
0503 08:13:55.177278 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.177280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.177480 (+ 200us) server_negotiation.cc:407] Connection header received
0503 08:13:55.177924 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.177925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.177932 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.178013 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.178842 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.178843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.178844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.178848 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.178865 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.178869 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.178943 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.179044 (+ 101us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.179064 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240503 08:13:55.179215 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 11131
I20240503 08:13:55.179270 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.177159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:36317 (local address 127.6.68.193:52937)
0503 08:13:55.177374 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:55.177401 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:55.177830 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:55.178061 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:55.178064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:55.178071 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:55.178706 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:13:55.178707 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:55.178726 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:13:55.178749 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:55.178940 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:55.178942 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:55.178944 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:13:55.179026 (+ 82us) client_negotiation.cc:240] Negotiation successful
0503 08:13:55.179080 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20240503 08:13:55.179332 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:13:55.179515 6419 external_mini_cluster.cc:1428] Setting key c44bdac91a78f43e855e635a84440b56
I20240503 08:13:55.180042 11259 heartbeater.cc:347] Connected to a master server at 127.6.68.254:36317
I20240503 08:13:55.180130 11259 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:55.180264 11259 heartbeater.cc:511] Master 127.6.68.254:36317 requested a full tablet report, sending...
I20240503 08:13:55.180532 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36317
--builtin_ntp_servers=127.6.68.212:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20240503 08:13:55.180753 11071 ts_manager.cc:194] Registered new tserver with Master: 81f97821481343a3b487dbb08707ad20 (127.6.68.193:39833)
W20240503 08:13:55.260255 11262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:55.260447 11262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:55.260470 11262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:55.261821 11262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:55.261873 11262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:55.261898 11262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:13:55.263351 11262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:55.263572 11262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:55.263839 11262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:55.268539 11268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.269203 11267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.269870 11270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:55.270823 11262 server_base.cc:1034] running on GCE node
I20240503 08:13:55.271009 11262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:55.271255 11262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:55.272392 11262 hybrid_clock.cc:648] HybridClock initialized: now 1714724035272386 us; error 31 us; skew 500 ppm
I20240503 08:13:55.273706 11262 webserver.cc:460] Webserver started at http://127.6.68.194:46017/ using document root <none> and password file <none>
I20240503 08:13:55.273901 11262 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:55.273948 11262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:55.274061 11262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:55.275075 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "4b72770f02a8449fbac63b09529f15ce"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "d0b8765855206ba8beb81436d06e61d3"
server_key_iv: "aa35422564e042a303383d3ce3d26352"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.275393 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "4b72770f02a8449fbac63b09529f15ce"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "d0b8765855206ba8beb81436d06e61d3"
server_key_iv: "aa35422564e042a303383d3ce3d26352"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.276639 11262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:13:55.277462 11276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.277652 11262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:13:55.277729 11262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "4b72770f02a8449fbac63b09529f15ce"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "d0b8765855206ba8beb81436d06e61d3"
server_key_iv: "aa35422564e042a303383d3ce3d26352"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.277797 11262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:55.281514 11262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:55.281731 11262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:55.281838 11262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:55.282001 11262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:55.282279 11262 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:55.282310 11262 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.282344 11262 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:55.282382 11262 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.288115 11262 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:38003
I20240503 08:13:55.288149 11389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:38003 every 8 connection(s)
I20240503 08:13:55.288496 11262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:13:55.290968 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.289268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:55317 (local address 127.6.68.254:36317)
0503 08:13:55.289336 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.289339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.289512 (+ 173us) server_negotiation.cc:407] Connection header received
0503 08:13:55.289971 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.289972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.289976 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.290034 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.290786 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.290786 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.290787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.290789 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.290799 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.290802 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.290838 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.290902 (+ 64us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.290913 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240503 08:13:55.291137 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.289209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:36317 (local address 127.6.68.194:55317)
0503 08:13:55.289408 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:55.289433 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:55.289882 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:55.290092 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:55.290096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:55.290104 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:55.290666 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:13:55.290666 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:55.290686 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:13:55.290709 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:55.290838 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:55.290841 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:55.290843 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:13:55.290919 (+ 76us) client_negotiation.cc:240] Negotiation successful
0503 08:13:55.290958 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20240503 08:13:55.291877 11390 heartbeater.cc:347] Connected to a master server at 127.6.68.254:36317
I20240503 08:13:55.291967 11390 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:55.292075 11390 heartbeater.cc:511] Master 127.6.68.254:36317 requested a full tablet report, sending...
I20240503 08:13:55.292421 11071 ts_manager.cc:194] Registered new tserver with Master: 4b72770f02a8449fbac63b09529f15ce (127.6.68.194:38003)
I20240503 08:13:55.294991 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 11262
I20240503 08:13:55.295087 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:13:55.295249 6419 external_mini_cluster.cc:1428] Setting key fa925c727f0a418294923e1cfa444bf9
I20240503 08:13:55.296226 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36317
--builtin_ntp_servers=127.6.68.212:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240503 08:13:55.375864 11393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:55.376072 11393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:55.376096 11393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:55.377451 11393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:55.377501 11393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:55.377526 11393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:13:55.378970 11393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:55.379174 11393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:55.379403 11393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:55.384645 11401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.384894 11399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.385811 11398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:55.386523 11393 server_base.cc:1034] running on GCE node
I20240503 08:13:55.386684 11393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:55.386898 11393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:55.388037 11393 hybrid_clock.cc:648] HybridClock initialized: now 1714724035388017 us; error 37 us; skew 500 ppm
I20240503 08:13:55.389405 11393 webserver.cc:460] Webserver started at http://127.6.68.195:41255/ using document root <none> and password file <none>
I20240503 08:13:55.389600 11393 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:55.389647 11393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:55.389763 11393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:55.390792 11393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "4f55bd3d98da4ad6a1f5836dd6867216"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "b7268f6ee4a6dbff3370daf8ceb90d4f"
server_key_iv: "fdfa6a7c6da5dc8137e52dc84f2ccd83"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.391099 11393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "4f55bd3d98da4ad6a1f5836dd6867216"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "b7268f6ee4a6dbff3370daf8ceb90d4f"
server_key_iv: "fdfa6a7c6da5dc8137e52dc84f2ccd83"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.392392 11393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:13:55.393066 11407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.393224 11393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:13:55.393302 11393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "4f55bd3d98da4ad6a1f5836dd6867216"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "b7268f6ee4a6dbff3370daf8ceb90d4f"
server_key_iv: "fdfa6a7c6da5dc8137e52dc84f2ccd83"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.393375 11393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:55.397008 11393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:55.397224 11393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:55.397328 11393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:55.397509 11393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:55.397787 11393 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:55.397820 11393 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.397853 11393 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:55.397872 11393 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.403177 11393 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:42667
I20240503 08:13:55.403227 11520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:42667 every 8 connection(s)
I20240503 08:13:55.403529 11393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:13:55.406033 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.404238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:54469 (local address 127.6.68.254:36317)
0503 08:13:55.404290 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.404292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.404506 (+ 214us) server_negotiation.cc:407] Connection header received
0503 08:13:55.404988 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.404989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.404996 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.405064 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.405780 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.405783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.405784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.405788 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.405802 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.405805 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.405857 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.405942 (+ 85us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.405958 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240503 08:13:55.406193 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.404204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:36317 (local address 127.6.68.195:54469)
0503 08:13:55.404401 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:55.404426 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:55.404880 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:55.405131 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:55.405135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:55.405142 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:55.405658 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:13:55.405658 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:55.405675 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:13:55.405696 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:55.405860 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:55.405864 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:55.405867 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:13:55.405967 (+ 100us) client_negotiation.cc:240] Negotiation successful
0503 08:13:55.406017 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20240503 08:13:55.406834 11521 heartbeater.cc:347] Connected to a master server at 127.6.68.254:36317
I20240503 08:13:55.406941 11521 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:55.407094 11521 heartbeater.cc:511] Master 127.6.68.254:36317 requested a full tablet report, sending...
I20240503 08:13:55.407514 11071 ts_manager.cc:194] Registered new tserver with Master: 4f55bd3d98da4ad6a1f5836dd6867216 (127.6.68.195:42667)
I20240503 08:13:55.410486 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 11393
I20240503 08:13:55.410573 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:13:55.410723 6419 external_mini_cluster.cc:1428] Setting key 9d0ca544ce8cf1d5195af0d2e4932765
I20240503 08:13:55.411506 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:13:55.413966 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.412738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.6.68.254:36317)
0503 08:13:55.412807 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.412809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.412879 (+ 70us) server_negotiation.cc:407] Connection header received
0503 08:13:55.412963 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.412966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.412971 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.413016 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.413710 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.413711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.413711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.413716 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.413735 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.413739 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.413810 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.413872 (+ 62us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.413890 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240503 08:13:55.415781 11071 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:13:55.417239 11071 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:13:55.420926 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.419052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:42667 (local address 127.0.0.1:37774)
0503 08:13:55.419184 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:55.419213 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:55.419280 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:55.419857 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:55.419858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:55.419861 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:55.420503 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:13:55.420503 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:55.420532 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:13:55.420556 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:55.420733 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:55.420737 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:55.420739 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:13:55.420814 (+ 75us) client_negotiation.cc:240] Negotiation successful
0503 08:13:55.420828 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20240503 08:13:55.420969 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.419159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.6.68.195:42667)
0503 08:13:55.419708 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.419716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.419725 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:13:55.419750 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.419751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.419758 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.420061 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.420630 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.420635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.420635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.420639 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.420673 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.420677 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.420739 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.420827 (+ 88us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.420840 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":89,"threads_started":1}
I20240503 08:13:55.421087 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.419451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39833 (local address 127.0.0.1:42958)
0503 08:13:55.419575 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:55.419584 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:55.419656 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:55.420121 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:55.420122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:55.420124 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:55.420671 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:13:55.420671 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:55.420680 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:13:55.420693 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:55.420952 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:55.420953 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:55.420953 (+ 0us) client_negotiation.cc:769] Sending connection context
0503 08:13:55.420983 (+ 30us) client_negotiation.cc:240] Negotiation successful
0503 08:13:55.420996 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20240503 08:13:55.421367 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.419683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.6.68.193:39833)
0503 08:13:55.419878 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.419887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.419896 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:13:55.419920 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.419922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.419930 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.420035 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.420783 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.420789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.420789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.420793 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.420843 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.420849 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.421211 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.421235 (+ 24us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.421249 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":102,"threads_started":1}
I20240503 08:13:55.421656 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.419241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38003 (local address 127.0.0.1:53326)
0503 08:13:55.419381 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:55.419391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:55.419436 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:55.419845 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:55.419850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:55.419856 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:55.420841 (+ 985us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:13:55.420841 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:55.420847 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:13:55.420857 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:55.421404 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:55.421405 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:55.421405 (+ 0us) client_negotiation.cc:769] Sending connection context
0503 08:13:55.421577 (+ 172us) client_negotiation.cc:240] Negotiation successful
0503 08:13:55.421592 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20240503 08:13:55.421626 11455 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5b34e55090443c8692b99dea1680e6 (DEFAULT_TABLE table=test-table [id=5866195541c840b7bd2d858a3b7fee6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:55.421944 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.419353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.6.68.194:38003)
0503 08:13:55.419507 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.419515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.419527 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:13:55.419550 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.419551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.419558 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.419656 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.420959 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.420966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.420967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.421263 (+ 296us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.421312 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.421320 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.421785 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.421811 (+ 26us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.421825 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20240503 08:13:55.421945 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5b34e55090443c8692b99dea1680e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:55.421891 11193 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5b34e55090443c8692b99dea1680e6 (DEFAULT_TABLE table=test-table [id=5866195541c840b7bd2d858a3b7fee6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:55.422188 11193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5b34e55090443c8692b99dea1680e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:55.422391 11324 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5b34e55090443c8692b99dea1680e6 (DEFAULT_TABLE table=test-table [id=5866195541c840b7bd2d858a3b7fee6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:55.422657 11324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5b34e55090443c8692b99dea1680e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:55.424571 11535 tablet_bootstrap.cc:492] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216: Bootstrap starting.
I20240503 08:13:55.425074 11535 tablet_bootstrap.cc:654] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:55.425361 11535 log.cc:824] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:55.425704 11536 tablet_bootstrap.cc:492] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20: Bootstrap starting.
I20240503 08:13:55.425714 11538 tablet_bootstrap.cc:492] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: Bootstrap starting.
I20240503 08:13:55.426095 11535 tablet_bootstrap.cc:492] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216: No bootstrap required, opened a new log
I20240503 08:13:55.426165 11535 ts_tablet_manager.cc:1367] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:55.426344 11538 tablet_bootstrap.cc:654] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:55.426580 11538 log.cc:824] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:55.427299 11538 tablet_bootstrap.cc:492] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: No bootstrap required, opened a new log
I20240503 08:13:55.427374 11538 ts_tablet_manager.cc:1367] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:55.428210 11536 tablet_bootstrap.cc:654] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:55.428294 11535 raft_consensus.cc:348] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 42667 } } peers { permanent_uuid: "4b72770f02a8449fbac63b09529f15ce" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38003 } } peers { permanent_uuid: "81f97821481343a3b487dbb08707ad20" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39833 } }
I20240503 08:13:55.428453 11535 raft_consensus.cc:374] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:55.428483 11535 raft_consensus.cc:729] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f55bd3d98da4ad6a1f5836dd6867216, State: Initialized, Role: FOLLOWER
I20240503 08:13:55.428462 11536 log.cc:824] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:55.428578 11535 consensus_queue.cc:260] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 42667 } } peers { permanent_uuid: "4b72770f02a8449fbac63b09529f15ce" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38003 } } peers { permanent_uuid: "81f97821481343a3b487dbb08707ad20" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39833 } }
I20240503 08:13:55.428817 11535 ts_tablet_manager.cc:1398] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:55.428874 11521 heartbeater.cc:503] Master 127.6.68.254:36317 was elected leader, sending a full tablet report...
I20240503 08:13:55.429373 11536 tablet_bootstrap.cc:492] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20: No bootstrap required, opened a new log
I20240503 08:13:55.429440 11536 ts_tablet_manager.cc:1367] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:13:55.429695 11538 raft_consensus.cc:348] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 42667 } } peers { permanent_uuid: "4b72770f02a8449fbac63b09529f15ce" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38003 } } peers { permanent_uuid: "81f97821481343a3b487dbb08707ad20" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39833 } }
I20240503 08:13:55.429806 11538 raft_consensus.cc:374] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:55.429832 11538 raft_consensus.cc:729] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b72770f02a8449fbac63b09529f15ce, State: Initialized, Role: FOLLOWER
I20240503 08:13:55.429939 11538 consensus_queue.cc:260] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 42667 } } peers { permanent_uuid: "4b72770f02a8449fbac63b09529f15ce" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38003 } } peers { permanent_uuid: "81f97821481343a3b487dbb08707ad20" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39833 } }
I20240503 08:13:55.430189 11538 ts_tablet_manager.cc:1398] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:13:55.430317 11390 heartbeater.cc:503] Master 127.6.68.254:36317 was elected leader, sending a full tablet report...
I20240503 08:13:55.431679 11536 raft_consensus.cc:348] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 42667 } } peers { permanent_uuid: "4b72770f02a8449fbac63b09529f15ce" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38003 } } peers { permanent_uuid: "81f97821481343a3b487dbb08707ad20" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39833 } }
I20240503 08:13:55.431828 11536 raft_consensus.cc:374] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:55.431850 11536 raft_consensus.cc:729] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f97821481343a3b487dbb08707ad20, State: Initialized, Role: FOLLOWER
I20240503 08:13:55.431974 11536 consensus_queue.cc:260] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 42667 } } peers { permanent_uuid: "4b72770f02a8449fbac63b09529f15ce" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38003 } } peers { permanent_uuid: "81f97821481343a3b487dbb08707ad20" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39833 } }
I20240503 08:13:55.432214 11536 ts_tablet_manager.cc:1398] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:55.432340 11259 heartbeater.cc:503] Master 127.6.68.254:36317 was elected leader, sending a full tablet report...
W20240503 08:13:55.539049 11391 tablet.cc:2369] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:13:55.539561 11543 raft_consensus.cc:482] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:55.539690 11543 raft_consensus.cc:504] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 42667 } } peers { permanent_uuid: "4b72770f02a8449fbac63b09529f15ce" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38003 } } peers { permanent_uuid: "81f97821481343a3b487dbb08707ad20" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39833 } }
I20240503 08:13:55.540040 11543 leader_election.cc:288] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f55bd3d98da4ad6a1f5836dd6867216 (127.6.68.195:42667), 4b72770f02a8449fbac63b09529f15ce (127.6.68.194:38003)
I20240503 08:13:55.541357 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.540231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:42667 (local address 127.6.68.193:41369)
0503 08:13:55.540289 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:55.540301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:55.540340 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:55.540504 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:55.540506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:55.540508 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:55.541029 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:13:55.541029 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:55.541040 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:13:55.541054 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:55.541245 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:55.541246 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:55.541247 (+ 1us) client_negotiation.cc:769] Sending connection context
0503 08:13:55.541292 (+ 45us) client_negotiation.cc:240] Negotiation successful
0503 08:13:55.541304 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20240503 08:13:55.541409 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.540308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:41369 (local address 127.6.68.195:42667)
0503 08:13:55.540361 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.540363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.540380 (+ 17us) server_negotiation.cc:407] Connection header received
0503 08:13:55.540408 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.540409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.540415 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.540483 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.541149 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.541151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.541151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.541156 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.541173 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.541177 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.541239 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.541309 (+ 70us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.541326 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240503 08:13:55.541651 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.540445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:39339 (local address 127.6.68.194:38003)
0503 08:13:55.540487 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.540489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.540640 (+ 151us) server_negotiation.cc:407] Connection header received
0503 08:13:55.540676 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.540677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.540682 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.540760 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.541411 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.541412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.541413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.541417 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.541433 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.541437 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.541495 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.541568 (+ 73us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.541586 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:13:55.541689 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.540357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38003 (local address 127.6.68.193:39339)
0503 08:13:55.540563 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:55.540570 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:55.540620 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:55.540766 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:55.540767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:55.540769 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:13:55.541293 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:13:55.541294 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:13:55.541301 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:13:55.541313 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:13:55.541520 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:13:55.541520 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:13:55.541521 (+ 1us) client_negotiation.cc:769] Sending connection context
0503 08:13:55.541590 (+ 69us) client_negotiation.cc:240] Negotiation successful
0503 08:13:55.541610 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":109,"threads_started":1}
I20240503 08:13:55.541895 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5b34e55090443c8692b99dea1680e6" candidate_uuid: "81f97821481343a3b487dbb08707ad20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" is_pre_election: true
I20240503 08:13:55.542007 11344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5b34e55090443c8692b99dea1680e6" candidate_uuid: "81f97821481343a3b487dbb08707ad20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b72770f02a8449fbac63b09529f15ce" is_pre_election: true
I20240503 08:13:55.542063 11475 raft_consensus.cc:2454] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f97821481343a3b487dbb08707ad20 in term 0.
I20240503 08:13:55.542119 11344 raft_consensus.cc:2454] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f97821481343a3b487dbb08707ad20 in term 0.
I20240503 08:13:55.542295 11147 leader_election.cc:302] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f55bd3d98da4ad6a1f5836dd6867216, 81f97821481343a3b487dbb08707ad20; no voters:
I20240503 08:13:55.542435 11543 raft_consensus.cc:2781] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:55.542507 11543 raft_consensus.cc:482] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:55.542529 11543 raft_consensus.cc:3037] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:55.543211 11543 raft_consensus.cc:504] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 42667 } } peers { permanent_uuid: "4b72770f02a8449fbac63b09529f15ce" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38003 } } peers { permanent_uuid: "81f97821481343a3b487dbb08707ad20" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39833 } }
I20240503 08:13:55.543319 11543 leader_election.cc:288] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [CANDIDATE]: Term 1 election: Requested vote from peers 4f55bd3d98da4ad6a1f5836dd6867216 (127.6.68.195:42667), 4b72770f02a8449fbac63b09529f15ce (127.6.68.194:38003)
I20240503 08:13:55.543570 11344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5b34e55090443c8692b99dea1680e6" candidate_uuid: "81f97821481343a3b487dbb08707ad20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b72770f02a8449fbac63b09529f15ce"
I20240503 08:13:55.543574 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5b34e55090443c8692b99dea1680e6" candidate_uuid: "81f97821481343a3b487dbb08707ad20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f55bd3d98da4ad6a1f5836dd6867216"
I20240503 08:13:55.543676 11475 raft_consensus.cc:3037] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:55.543676 11344 raft_consensus.cc:3037] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:55.544564 11475 raft_consensus.cc:2454] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f97821481343a3b487dbb08707ad20 in term 1.
I20240503 08:13:55.544564 11344 raft_consensus.cc:2454] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f97821481343a3b487dbb08707ad20 in term 1.
I20240503 08:13:55.544754 11147 leader_election.cc:302] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f55bd3d98da4ad6a1f5836dd6867216, 81f97821481343a3b487dbb08707ad20; no voters:
I20240503 08:13:55.544852 11543 raft_consensus.cc:2781] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:55.545025 11543 raft_consensus.cc:686] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 1 LEADER]: Becoming Leader. State: Replica: 81f97821481343a3b487dbb08707ad20, State: Running, Role: LEADER
I20240503 08:13:55.545125 11543 consensus_queue.cc:237] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 42667 } } peers { permanent_uuid: "4b72770f02a8449fbac63b09529f15ce" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38003 } } peers { permanent_uuid: "81f97821481343a3b487dbb08707ad20" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39833 } }
I20240503 08:13:55.545997 11071 catalog_manager.cc:5484] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81f97821481343a3b487dbb08707ad20 (127.6.68.193). New cstate: current_term: 1 leader_uuid: "81f97821481343a3b487dbb08707ad20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b72770f02a8449fbac63b09529f15ce" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81f97821481343a3b487dbb08707ad20" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39833 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:55.570796 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.569561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.6.68.193:39833)
0503 08:13:55.569629 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.569631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.569641 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:13:55.569679 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.569680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.569691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.569761 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.570501 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.570502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.570502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.570509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.570538 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.570541 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.570618 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.570706 (+ 88us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.570721 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240503 08:13:55.572697 11475 raft_consensus.cc:1261] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216 [term 1 FOLLOWER]: Refusing update from remote peer 81f97821481343a3b487dbb08707ad20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:55.572697 11344 raft_consensus.cc:1261] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce [term 1 FOLLOWER]: Refusing update from remote peer 81f97821481343a3b487dbb08707ad20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:55.573043 11543 consensus_queue.cc:1035] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f55bd3d98da4ad6a1f5836dd6867216" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:55.573184 11545 consensus_queue.cc:1035] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b72770f02a8449fbac63b09529f15ce" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:55.575973 11550 mvcc.cc:205] Tried to move back new op lower bound from 7023509649703145472 to 7023509649593577472. Current Snapshot: MvccSnapshot[applied={T|T < 7023509649703145472 or (T in {7023509649703145472})}]
I20240503 08:13:55.576176 11549 mvcc.cc:205] Tried to move back new op lower bound from 7023509649703145472 to 7023509649593577472. Current Snapshot: MvccSnapshot[applied={T|T < 7023509649703145472}]
I20240503 08:13:55.577725 11071 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40198:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:55.577826 11071 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40198:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:13:55.578606 11071 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c0d91df0a8b64878a700e348a0790927: Sending DeleteTablet for 3 replicas of tablet 1a5b34e55090443c8692b99dea1680e6
I20240503 08:13:55.578948 11455 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5b34e55090443c8692b99dea1680e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:55 UTC) from {username='slave'} at 127.0.0.1:37774
I20240503 08:13:55.579082 11193 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5b34e55090443c8692b99dea1680e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:55 UTC) from {username='slave'} at 127.0.0.1:42958
I20240503 08:13:55.579174 11557 tablet_replica.cc:330] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216: stopping tablet replica
I20240503 08:13:55.579252 11557 raft_consensus.cc:2229] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:55.579304 11324 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5b34e55090443c8692b99dea1680e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:13:55 UTC) from {username='slave'} at 127.0.0.1:53326
I20240503 08:13:55.579339 11557 raft_consensus.cc:2258] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:55.579514 11559 tablet_replica.cc:330] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: stopping tablet replica
I20240503 08:13:55.579654 11559 raft_consensus.cc:2229] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:55.579766 11558 tablet_replica.cc:330] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20: stopping tablet replica
I20240503 08:13:55.579804 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 11131
I20240503 08:13:55.579843 11558 raft_consensus.cc:2229] T 1a5b34e55090443c8692b99dea1680e6 P 81f97821481343a3b487dbb08707ad20 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:13:55.580399 11559 raft_consensus.cc:2258] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:55.580899 11559 ts_tablet_manager.cc:1875] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:55.581723 11557 ts_tablet_manager.cc:1875] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:13:55.582540 11559 ts_tablet_manager.cc:1888] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:13:55.582602 11559 log.cc:1196] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/1a5b34e55090443c8692b99dea1680e6
I20240503 08:13:55.582862 11559 ts_tablet_manager.cc:1909] T 1a5b34e55090443c8692b99dea1680e6 P 4b72770f02a8449fbac63b09529f15ce: Deleting consensus metadata
I20240503 08:13:55.583284 11056 catalog_manager.cc:4830] TS 4b72770f02a8449fbac63b09529f15ce (127.6.68.194:38003): tablet 1a5b34e55090443c8692b99dea1680e6 (table test-table [id=5866195541c840b7bd2d858a3b7fee6c]) successfully deleted
I20240503 08:13:55.583598 11557 ts_tablet_manager.cc:1888] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:13:55.583699 11557 log.cc:1196] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/1a5b34e55090443c8692b99dea1680e6
I20240503 08:13:55.583951 11557 ts_tablet_manager.cc:1909] T 1a5b34e55090443c8692b99dea1680e6 P 4f55bd3d98da4ad6a1f5836dd6867216: Deleting consensus metadata
I20240503 08:13:55.584316 11056 catalog_manager.cc:4830] TS 4f55bd3d98da4ad6a1f5836dd6867216 (127.6.68.195:42667): tablet 1a5b34e55090443c8692b99dea1680e6 (table test-table [id=5866195541c840b7bd2d858a3b7fee6c]) successfully deleted
W20240503 08:13:55.585844 11056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:39833 (error 108)
I20240503 08:13:55.586095 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 11262
I20240503 08:13:55.586298 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.586165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39833 (local address 127.0.0.1:42976)
0503 08:13:55.586227 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:55.586257 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:39833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20240503 08:13:55.587574 11056 catalog_manager.cc:4573] TS 81f97821481343a3b487dbb08707ad20 (127.6.68.193:39833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a5b34e55090443c8692b99dea1680e6: Network error: Client connection negotiation failed: client connection to 127.6.68.193:39833: connect: Connection refused (error 111)
I20240503 08:13:55.591012 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 11393
I20240503 08:13:55.595326 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 11040
2024-05-03T08:13:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (721 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20240503 08:13:55.614698 6419 mini_oidc.cc:152] Starting JWKS server
I20240503 08:13:55.615737 6419 webserver.cc:460] Webserver started at https://127.0.0.1:33281/ using document root <none> and password file <none>
I20240503 08:13:55.615821 6419 mini_oidc.cc:189] Starting OIDC Discovery server
I20240503 08:13:55.615998 6419 webserver.cc:460] Webserver started at http://127.0.0.1:39589/ using document root <none> and password file <none>
2024-05-03T08:13:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:55Z Disabled control of system clock
I20240503 08:13:55.622627 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:41891
--webserver_interface=127.6.68.254
--webserver_port=0
--builtin_ntp_servers=127.6.68.212:46775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:41891
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33281/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/testchainca.cert with env {}
W20240503 08:13:55.703295 11566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:55.703495 11566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:55.703529 11566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:55.704856 11566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:55.704902 11566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:13:55.704923 11566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:55.704941 11566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:55.704959 11566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240503 08:13:55.704996 11566 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240503 08:13:55.705025 11566 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33281/jwks.json
I20240503 08:13:55.706683 11566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:46775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:41891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:41891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33281/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:13:55.706907 11566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:55.707079 11566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:55.711861 11574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.712476 11571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:55.713398 11572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:55.714118 11566 server_base.cc:1034] running on GCE node
I20240503 08:13:55.714329 11566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:55.714547 11566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:55.715685 11566 hybrid_clock.cc:648] HybridClock initialized: now 1714724035715666 us; error 36 us; skew 500 ppm
I20240503 08:13:55.716895 11566 webserver.cc:460] Webserver started at http://127.6.68.254:41343/ using document root <none> and password file <none>
I20240503 08:13:55.717087 11566 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:55.717137 11566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:55.717247 11566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:55.718235 11566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "562a9bfe4a74474795a59b775b60e80e"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "311b4c7f8df351647af0ae244dcd44ca"
server_key_iv: "667844b1e2053ae786e7f6c57c91ab45"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.718546 11566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "562a9bfe4a74474795a59b775b60e80e"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "311b4c7f8df351647af0ae244dcd44ca"
server_key_iv: "667844b1e2053ae786e7f6c57c91ab45"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.719738 11566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240503 08:13:55.720489 11580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:55.720649 11566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:13:55.720726 11566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "562a9bfe4a74474795a59b775b60e80e"
format_stamp: "Formatted at 2024-05-03 08:13:55 on dist-test-slave-3g67"
server_key: "311b4c7f8df351647af0ae244dcd44ca"
server_key_iv: "667844b1e2053ae786e7f6c57c91ab45"
server_key_version: "encryptionkey@0"
I20240503 08:13:55.720789 11566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:55.734520 11566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:55.734823 11566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:55.734939 11566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:55.738785 11566 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:41891
I20240503 08:13:55.738818 11632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:41891 every 8 connection(s)
I20240503 08:13:55.739140 11566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:13:55.739764 11633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:55.742198 11633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e: Bootstrap starting.
I20240503 08:13:55.742731 11633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:55.742959 11633 log.cc:824] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:55.743589 11633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e: No bootstrap required, opened a new log
I20240503 08:13:55.744987 11633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562a9bfe4a74474795a59b775b60e80e" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41891 } }
I20240503 08:13:55.745110 11633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:55.745126 11633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562a9bfe4a74474795a59b775b60e80e, State: Initialized, Role: FOLLOWER
I20240503 08:13:55.745227 11633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562a9bfe4a74474795a59b775b60e80e" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41891 } }
I20240503 08:13:55.745297 11633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:55.745321 11633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:55.745345 11633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:55.745834 11633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562a9bfe4a74474795a59b775b60e80e" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41891 } }
I20240503 08:13:55.745927 11633 leader_election.cc:302] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 562a9bfe4a74474795a59b775b60e80e; no voters:
I20240503 08:13:55.746042 11633 leader_election.cc:288] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:55.746126 11636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:55.746241 11633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:55.746287 11636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [term 1 LEADER]: Becoming Leader. State: Replica: 562a9bfe4a74474795a59b775b60e80e, State: Running, Role: LEADER
I20240503 08:13:55.746376 11636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562a9bfe4a74474795a59b775b60e80e" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41891 } }
I20240503 08:13:55.746608 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 11566
I20240503 08:13:55.746713 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:13:55.747009 6419 external_mini_cluster.cc:1428] Setting key 1b316655a7d97b4e50da840e67e76ee0
I20240503 08:13:55.747941 11638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 562a9bfe4a74474795a59b775b60e80e. Latest consensus state: current_term: 1 leader_uuid: "562a9bfe4a74474795a59b775b60e80e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562a9bfe4a74474795a59b775b60e80e" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41891 } } }
I20240503 08:13:55.748054 11638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:55.748270 11653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:55.747980 11637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "562a9bfe4a74474795a59b775b60e80e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562a9bfe4a74474795a59b775b60e80e" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41891 } } }
I20240503 08:13:55.748790 11637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:55.748736 11653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:55.750133 11653 catalog_manager.cc:1347] Generated new cluster ID: ab09f6e11d68463ca76fc2fd8b8c8add
I20240503 08:13:55.750190 11653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:13:55.752856 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.747514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.6.68.254:41891)
0503 08:13:55.747936 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.747943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.748125 (+ 182us) server_negotiation.cc:407] Connection header received
0503 08:13:55.748347 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.748353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.748413 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.748536 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:13:55.749135 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:55.750302 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:55.751108 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:55.751334 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:55.752171 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:13:55.752197 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:13:55.752198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:13:55.752407 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:13:55.752437 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:13:55.752445 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:13:55.752491 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.752632 (+ 141us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.752690 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":69,"threads_started":1}
I20240503 08:13:55.753787 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:13:55.765918 11653 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:55.766664 11653 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:55.777308 11653 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 562a9bfe4a74474795a59b775b60e80e: Generated new TSK 0
I20240503 08:13:55.777529 11653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:13:55.809142 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.788659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.6.68.254:41891)
0503 08:13:55.788711 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.788714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.788888 (+ 174us) server_negotiation.cc:407] Connection header received
0503 08:13:55.788999 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.789000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.789013 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.789110 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0503 08:13:55.789546 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:55.790359 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:55.790988 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:55.791171 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:55.791224 (+ 53us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0503 08:13:55.808815 (+ 17591us) server_negotiation.cc:377] Sending JWT_EXCHANGE NegotiatePB response
0503 08:13:55.808901 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0503 08:13:55.809015 (+ 114us) server_negotiation.cc:299] Negotiation successful
0503 08:13:55.809048 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26,"thread_start_us":128,"threads_started":1}
I20240503 08:13:55.816550 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:13:55.813926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.6.68.254:41891)
0503 08:13:55.813969 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:13:55.813971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:13:55.814048 (+ 77us) server_negotiation.cc:407] Connection header received
0503 08:13:55.814104 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:13:55.814105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:13:55.814114 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:13:55.814172 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0503 08:13:55.814395 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:55.815154 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:13:55.815712 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:13:55.815889 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:55.815913 (+ 24us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0503 08:13:55.816339 (+ 426us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0503 08:13:55.816486 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52186: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":22}
W20240503 08:13:55.816622 11652 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52186: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20240503 08:13:55.816673 11669 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:13:55.813863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:41891 (local address 127.0.0.1:52186)
0503 08:13:55.813977 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:55.813987 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:55.814046 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:55.814174 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:13:55.814175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:13:55.814176 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0503 08:13:55.814333 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:55.814334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:55.815235 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:13:55.815235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:13:55.815612 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:13:55.815614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:13:55.815705 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:13:55.815710 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0503 08:13:55.816452 (+ 742us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0503 08:13:55.816518 (+ 66us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.68.254:41891: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":72,"threads_started":1}
I20240503 08:14:10.823139 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:10.821124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.6.68.254:41891)
0503 08:14:10.821247 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:10.821249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:10.821269 (+ 20us) server_negotiation.cc:407] Connection header received
0503 08:14:10.821347 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:10.821347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:10.821358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:10.821411 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0503 08:14:10.821649 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:10.822154 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:10.822695 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:10.822808 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:10.822834 (+ 26us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0503 08:14:10.822998 (+ 164us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0503 08:14:10.823086 (+ 88us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32882: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
W20240503 08:14:10.823210 11675 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32882: JWT verification failed: token expired
W20240503 08:14:10.823231 11674 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:10.820953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:41891 (local address 127.0.0.1:32882)
0503 08:14:10.821173 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:10.821184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:10.821275 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:10.821421 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:10.821422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:10.821424 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0503 08:14:10.821592 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:10.821594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:10.822213 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:10.822213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:10.822631 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:10.822632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:10.822689 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:10.822694 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0503 08:14:10.823068 (+ 374us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0503 08:14:10.823169 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.68.254:41891: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":157,"threads_started":1}
W20240503 08:14:10.824575 11675 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:10.824216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.6.68.254:41891)
0503 08:14:10.824274 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:10.824276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:10.824335 (+ 59us) server_negotiation.cc:407] Connection header received
0503 08:14:10.824373 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:10.824373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:10.824383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:10.824433 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:10.824532 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32890: BlockingRecv error: recv got EOF from 127.0.0.1:32890 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20240503 08:14:10.827242 11685 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.6.68.254:41891
W20240503 08:14:10.827562 11675 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:10.827100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.6.68.254:41891)
0503 08:14:10.827152 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:10.827155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:10.827218 (+ 63us) server_negotiation.cc:407] Connection header received
0503 08:14:10.827363 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:10.827364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:10.827374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:10.827427 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:10.827526 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32894: BlockingRecv error: recv got EOF from 127.0.0.1:32894 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:14:10.827708 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 11566
2024-05-03T08:14:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15251 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20240503 08:14:10.866070 6419 mini_oidc.cc:152] Starting JWKS server
I20240503 08:14:10.866820 6419 webserver.cc:460] Webserver started at https://127.0.0.1:42485/ using document root <none> and password file <none>
I20240503 08:14:10.866865 6419 mini_oidc.cc:189] Starting OIDC Discovery server
I20240503 08:14:10.866986 6419 webserver.cc:460] Webserver started at http://127.0.0.1:36069/ using document root <none> and password file <none>
2024-05-03T08:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:14:10Z Disabled control of system clock
I20240503 08:14:10.873613 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:39827
--webserver_interface=127.6.68.254
--webserver_port=0
--builtin_ntp_servers=127.6.68.212:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:39827
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42485/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/testchainca.cert with env {}
W20240503 08:14:10.953464 11691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:10.953672 11691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:10.953694 11691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:10.955070 11691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:10.955128 11691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:10.955140 11691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:10.955152 11691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:10.955164 11691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240503 08:14:10.955176 11691 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240503 08:14:10.955188 11691 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42485/jwks.json
I20240503 08:14:10.956993 11691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:39827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:39827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42485/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:10.957233 11691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:10.957409 11691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:14:10.962565 11691 server_base.cc:1034] running on GCE node
W20240503 08:14:10.963013 11696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:10.963693 11697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:10.964494 11699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:10.964893 11691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:10.965270 11691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:10.966436 11691 hybrid_clock.cc:648] HybridClock initialized: now 1714724050966420 us; error 35 us; skew 500 ppm
I20240503 08:14:10.972641 11691 webserver.cc:460] Webserver started at http://127.6.68.254:33689/ using document root <none> and password file <none>
I20240503 08:14:10.972885 11691 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:10.972942 11691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:10.973073 11691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:10.974138 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "17de27542b64469184a07241814ba0c4"
format_stamp: "Formatted at 2024-05-03 08:14:10 on dist-test-slave-3g67"
server_key: "a2c81bcb1ddcac1e06203f27094f2e2f"
server_key_iv: "23f57b828f34912efb774aaf78cedba5"
server_key_version: "encryptionkey@0"
I20240503 08:14:10.974473 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "17de27542b64469184a07241814ba0c4"
format_stamp: "Formatted at 2024-05-03 08:14:10 on dist-test-slave-3g67"
server_key: "a2c81bcb1ddcac1e06203f27094f2e2f"
server_key_iv: "23f57b828f34912efb774aaf78cedba5"
server_key_version: "encryptionkey@0"
I20240503 08:14:10.975831 11691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240503 08:14:10.976634 11705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:10.976853 11691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:10.976946 11691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "17de27542b64469184a07241814ba0c4"
format_stamp: "Formatted at 2024-05-03 08:14:10 on dist-test-slave-3g67"
server_key: "a2c81bcb1ddcac1e06203f27094f2e2f"
server_key_iv: "23f57b828f34912efb774aaf78cedba5"
server_key_version: "encryptionkey@0"
I20240503 08:14:10.977030 11691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:10.992415 11691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:10.992719 11691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:10.992839 11691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:10.996521 11691 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:39827
I20240503 08:14:10.996563 11757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:39827 every 8 connection(s)
I20240503 08:14:10.996860 11691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:14:10.997476 11758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:10.997844 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 11691
I20240503 08:14:10.997956 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:14:10.998121 6419 external_mini_cluster.cc:1428] Setting key 88e231e137f686342c0a150d23650405
I20240503 08:14:11.001040 11758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4: Bootstrap starting.
I20240503 08:14:11.001711 11758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:11.002022 11758 log.cc:824] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:11.002873 11758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4: No bootstrap required, opened a new log
I20240503 08:14:11.003067 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:10.998672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.6.68.254:39827)
0503 08:14:10.998853 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:10.998859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:10.998878 (+ 19us) server_negotiation.cc:407] Connection header received
0503 08:14:10.999458 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:10.999467 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:10.999526 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:10.999715 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:11.000115 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.000829 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:11.001651 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.001917 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:11.002415 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:11.002424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:11.002425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:14:11.002617 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:11.002646 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:14:11.002654 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:11.002712 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:11.002834 (+ 122us) server_negotiation.cc:299] Negotiation successful
0503 08:14:11.002879 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20240503 08:14:11.003852 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:14:11.004699 11758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17de27542b64469184a07241814ba0c4" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 39827 } }
I20240503 08:14:11.004825 11758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:11.004856 11758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17de27542b64469184a07241814ba0c4, State: Initialized, Role: FOLLOWER
I20240503 08:14:11.004947 11758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17de27542b64469184a07241814ba0c4" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 39827 } }
I20240503 08:14:11.005003 11758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:14:11.005043 11758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:14:11.005093 11758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:11.005653 11758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17de27542b64469184a07241814ba0c4" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 39827 } }
I20240503 08:14:11.005748 11758 leader_election.cc:302] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17de27542b64469184a07241814ba0c4; no voters:
I20240503 08:14:11.005930 11758 leader_election.cc:288] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:14:11.005996 11763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:11.006142 11758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:11.006157 11763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [term 1 LEADER]: Becoming Leader. State: Replica: 17de27542b64469184a07241814ba0c4, State: Running, Role: LEADER
I20240503 08:14:11.006359 11763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17de27542b64469184a07241814ba0c4" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 39827 } }
I20240503 08:14:11.006719 11764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17de27542b64469184a07241814ba0c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17de27542b64469184a07241814ba0c4" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 39827 } } }
I20240503 08:14:11.006811 11764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:11.006981 11771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:14:11.007028 11765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17de27542b64469184a07241814ba0c4. Latest consensus state: current_term: 1 leader_uuid: "17de27542b64469184a07241814ba0c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17de27542b64469184a07241814ba0c4" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 39827 } } }
I20240503 08:14:11.007138 11765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:11.007485 11771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:14:11.008989 11771 catalog_manager.cc:1347] Generated new cluster ID: c7822652c6184a4a993537fd87a800e3
I20240503 08:14:11.009048 11771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:14:11.015697 11771 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:14:11.016244 11771 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:14:11.023844 11771 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 17de27542b64469184a07241814ba0c4: Generated new TSK 0
I20240503 08:14:11.024048 11771 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240503 08:14:11.049885 11788 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240503 08:14:11.049981 11788 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20240503 08:14:11.050360 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:11.034879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.6.68.254:39827)
0503 08:14:11.034930 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:11.034932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:11.035009 (+ 77us) server_negotiation.cc:407] Connection header received
0503 08:14:11.035138 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:11.035139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:11.035151 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:11.035269 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0503 08:14:11.035602 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.036535 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:11.037413 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.037554 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:11.037604 (+ 50us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0503 08:14:11.050100 (+ 12496us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0503 08:14:11.050284 (+ 184us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48586: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":29}
W20240503 08:14:11.050423 11761 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48586: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20240503 08:14:11.050424 11787 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:11.034786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:39827 (local address 127.0.0.1:48586)
0503 08:14:11.034926 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:11.034938 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:11.035020 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:11.035273 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:11.035273 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:11.035276 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0503 08:14:11.035522 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:11.035524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.036642 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:11.036643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:11.037323 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:11.037326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.037412 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:11.037418 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0503 08:14:11.050219 (+ 12801us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0503 08:14:11.050332 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.68.254:39827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20240503 08:14:11.050732 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 11691
2024-05-03T08:14:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (222 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20240503 08:14:11.088816 6419 mini_oidc.cc:152] Starting JWKS server
I20240503 08:14:11.089577 6419 webserver.cc:460] Webserver started at https://127.0.0.1:41331/ using document root <none> and password file <none>
I20240503 08:14:11.089619 6419 mini_oidc.cc:189] Starting OIDC Discovery server
I20240503 08:14:11.089731 6419 webserver.cc:460] Webserver started at http://127.0.0.1:40893/ using document root <none> and password file <none>
2024-05-03T08:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:14:11Z Disabled control of system clock
I20240503 08:14:11.096560 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:38427
--webserver_interface=127.6.68.254
--webserver_port=0
--builtin_ntp_servers=127.6.68.212:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:38427
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41331/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240503 08:14:11.177029 11794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:11.177246 11794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:11.177281 11794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:11.178601 11794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:11.178651 11794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:11.178670 11794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:11.178690 11794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:11.178709 11794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240503 08:14:11.178728 11794 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240503 08:14:11.178762 11794 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41331/jwks.json
I20240503 08:14:11.180297 11794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:38427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:38427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41331/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:11.180544 11794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:11.180791 11794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:11.185330 11802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:11.185887 11800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:11.186599 11799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:11.187580 11794 server_base.cc:1034] running on GCE node
I20240503 08:14:11.187855 11794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:11.188230 11794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:11.189414 11794 hybrid_clock.cc:648] HybridClock initialized: now 1714724051189395 us; error 41 us; skew 500 ppm
I20240503 08:14:11.190791 11794 webserver.cc:460] Webserver started at http://127.6.68.254:35601/ using document root <none> and password file <none>
I20240503 08:14:11.190980 11794 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:11.191025 11794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:11.191138 11794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:11.192188 11794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "9ee2293d498e4ec8aae48ce933a48bd1"
format_stamp: "Formatted at 2024-05-03 08:14:11 on dist-test-slave-3g67"
server_key: "68147fd9128d99542557acd7f250c7d3"
server_key_iv: "5a3e7f09ea0384f901393b2a362f4cf7"
server_key_version: "encryptionkey@0"
I20240503 08:14:11.192482 11794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "9ee2293d498e4ec8aae48ce933a48bd1"
format_stamp: "Formatted at 2024-05-03 08:14:11 on dist-test-slave-3g67"
server_key: "68147fd9128d99542557acd7f250c7d3"
server_key_iv: "5a3e7f09ea0384f901393b2a362f4cf7"
server_key_version: "encryptionkey@0"
I20240503 08:14:11.193694 11794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:14:11.194489 11808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:11.194702 11794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:11.194797 11794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "9ee2293d498e4ec8aae48ce933a48bd1"
format_stamp: "Formatted at 2024-05-03 08:14:11 on dist-test-slave-3g67"
server_key: "68147fd9128d99542557acd7f250c7d3"
server_key_iv: "5a3e7f09ea0384f901393b2a362f4cf7"
server_key_version: "encryptionkey@0"
I20240503 08:14:11.194856 11794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:11.226469 11794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:11.226812 11794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:11.226926 11794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:11.230839 11794 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:38427
I20240503 08:14:11.230896 11860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:38427 every 8 connection(s)
I20240503 08:14:11.231194 11794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:14:11.231858 11861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:11.234364 11861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1: Bootstrap starting.
I20240503 08:14:11.234928 11861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:11.235179 11861 log.cc:824] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:11.236006 11861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1: No bootstrap required, opened a new log
I20240503 08:14:11.237526 11861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee2293d498e4ec8aae48ce933a48bd1" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38427 } }
I20240503 08:14:11.237658 11861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:11.237679 11861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee2293d498e4ec8aae48ce933a48bd1, State: Initialized, Role: FOLLOWER
I20240503 08:14:11.237810 11861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee2293d498e4ec8aae48ce933a48bd1" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38427 } }
I20240503 08:14:11.237869 11861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:14:11.237890 11861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:14:11.237937 11861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:11.238541 11861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee2293d498e4ec8aae48ce933a48bd1" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38427 } }
I20240503 08:14:11.238632 11861 leader_election.cc:302] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ee2293d498e4ec8aae48ce933a48bd1; no voters:
I20240503 08:14:11.238798 11861 leader_election.cc:288] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:14:11.238895 11864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:11.239006 11861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:11.239066 11864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [term 1 LEADER]: Becoming Leader. State: Replica: 9ee2293d498e4ec8aae48ce933a48bd1, State: Running, Role: LEADER
I20240503 08:14:11.239190 11864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee2293d498e4ec8aae48ce933a48bd1" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38427 } }
I20240503 08:14:11.239531 11866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ee2293d498e4ec8aae48ce933a48bd1. Latest consensus state: current_term: 1 leader_uuid: "9ee2293d498e4ec8aae48ce933a48bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee2293d498e4ec8aae48ce933a48bd1" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38427 } } }
I20240503 08:14:11.239723 11866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:11.239535 11865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ee2293d498e4ec8aae48ce933a48bd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee2293d498e4ec8aae48ce933a48bd1" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38427 } } }
I20240503 08:14:11.240003 11873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:14:11.240655 11873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:14:11.240938 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 11794
I20240503 08:14:11.241035 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:14:11.240923 11865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:11.241245 6419 external_mini_cluster.cc:1428] Setting key 423e55f338a7b37e0f7d86fdd87aedf9
I20240503 08:14:11.242810 11873 catalog_manager.cc:1347] Generated new cluster ID: a2d6481c86b449ca8dbee53cc32a5d17
I20240503 08:14:11.242882 11873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:14:11.251835 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:11.241746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.6.68.254:38427)
0503 08:14:11.241966 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:11.241973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:11.242031 (+ 58us) server_negotiation.cc:407] Connection header received
0503 08:14:11.242174 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:11.242180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:11.242248 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:11.242399 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:11.248076 (+ 5677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.249415 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:11.250239 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.250498 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:11.251100 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:11.251109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:11.251109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:14:11.251333 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:11.251370 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:14:11.251379 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:11.251462 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:11.251532 (+ 70us) server_negotiation.cc:299] Negotiation successful
0503 08:14:11.251596 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20240503 08:14:11.252954 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:14:11.266919 11873 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:14:11.267477 11873 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:14:11.274168 11873 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9ee2293d498e4ec8aae48ce933a48bd1: Generated new TSK 0
I20240503 08:14:11.274394 11873 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240503 08:14:11.299691 11891 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240503 08:14:11.299769 11891 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20240503 08:14:11.301056 11890 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:11.283787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:38427 (local address 127.0.0.1:59512)
0503 08:14:11.283951 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:11.283961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:11.284020 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:11.284202 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:11.284203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:11.284205 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0503 08:14:11.284379 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:11.284381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.285419 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:11.285421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:11.285864 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:11.285866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.285930 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:11.285935 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0503 08:14:11.300881 (+ 14946us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41331/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0503 08:14:11.300983 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.68.254:38427: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41331/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20240503 08:14:11.301101 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:11.283884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.6.68.254:38427)
0503 08:14:11.283931 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:11.283933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:11.284019 (+ 86us) server_negotiation.cc:407] Connection header received
0503 08:14:11.284087 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:11.284088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:11.284103 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:11.284199 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0503 08:14:11.284444 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.285297 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:11.285942 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.286134 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:11.286196 (+ 62us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0503 08:14:11.300754 (+ 14558us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41331/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0503 08:14:11.301027 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59512: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41331/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20240503 08:14:11.301154 11883 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59512: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41331/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20240503 08:14:11.301280 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 11794
2024-05-03T08:14:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (250 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2024-05-03T08:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:14:11Z Disabled control of system clock
I20240503 08:14:11.346330 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:36429
--webserver_interface=127.6.68.254
--webserver_port=0
--builtin_ntp_servers=127.6.68.212:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:36429
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/testchainca.cert with env {}
W20240503 08:14:11.427567 11895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:11.427816 11895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:11.427862 11895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:11.429188 11895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:11.429239 11895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:11.429258 11895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:11.429277 11895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:11.429294 11895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240503 08:14:11.429311 11895 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240503 08:14:11.429327 11895 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20240503 08:14:11.431041 11895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:36429
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:36429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:11.431285 11895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:11.431469 11895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:11.436277 11903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:11.436894 11900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:11.437491 11901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:11.438361 11895 server_base.cc:1034] running on GCE node
I20240503 08:14:11.438628 11895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:11.438905 11895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:11.440059 11895 hybrid_clock.cc:648] HybridClock initialized: now 1714724051440032 us; error 48 us; skew 500 ppm
I20240503 08:14:11.441534 11895 webserver.cc:460] Webserver started at http://127.6.68.254:33181/ using document root <none> and password file <none>
I20240503 08:14:11.441735 11895 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:11.441782 11895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:11.441892 11895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:11.442924 11895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "2061429195ca44b68f0dc26b2dc82deb"
format_stamp: "Formatted at 2024-05-03 08:14:11 on dist-test-slave-3g67"
server_key: "df0b0b5365b4a23bb085e0c0f64e9bef"
server_key_iv: "3d830eb7d4d3f0ad5dd4d7594c1d59b6"
server_key_version: "encryptionkey@0"
I20240503 08:14:11.443239 11895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "2061429195ca44b68f0dc26b2dc82deb"
format_stamp: "Formatted at 2024-05-03 08:14:11 on dist-test-slave-3g67"
server_key: "df0b0b5365b4a23bb085e0c0f64e9bef"
server_key_iv: "3d830eb7d4d3f0ad5dd4d7594c1d59b6"
server_key_version: "encryptionkey@0"
I20240503 08:14:11.444643 11895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:14:11.445326 11909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:11.445509 11895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:14:11.445592 11895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "2061429195ca44b68f0dc26b2dc82deb"
format_stamp: "Formatted at 2024-05-03 08:14:11 on dist-test-slave-3g67"
server_key: "df0b0b5365b4a23bb085e0c0f64e9bef"
server_key_iv: "3d830eb7d4d3f0ad5dd4d7594c1d59b6"
server_key_version: "encryptionkey@0"
I20240503 08:14:11.445657 11895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:11.454447 11895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:11.454725 11895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:11.454838 11895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:11.458478 11895 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:36429
I20240503 08:14:11.458544 11961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:36429 every 8 connection(s)
I20240503 08:14:11.458853 11895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:14:11.459475 11962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:11.460565 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 11895
I20240503 08:14:11.460686 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:14:11.460911 6419 external_mini_cluster.cc:1428] Setting key f52121794f9e88119aafcaeadc64b1c5
I20240503 08:14:11.462838 11962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb: Bootstrap starting.
I20240503 08:14:11.463382 11962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:11.463613 11962 log.cc:824] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:11.464330 11962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb: No bootstrap required, opened a new log
I20240503 08:14:11.465263 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:11.461437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.6.68.254:36429)
0503 08:14:11.461603 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:11.461609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:11.461628 (+ 19us) server_negotiation.cc:407] Connection header received
0503 08:14:11.462020 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:11.462026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:11.462079 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:11.462197 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:11.462630 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.463416 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:11.464072 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.464324 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:11.464755 (+ 431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:11.464763 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:11.464763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:14:11.464913 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:11.464937 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:14:11.464944 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:11.465006 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:11.465084 (+ 78us) server_negotiation.cc:299] Negotiation successful
0503 08:14:11.465136 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20240503 08:14:11.466169 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:14:11.466248 11962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2061429195ca44b68f0dc26b2dc82deb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36429 } }
I20240503 08:14:11.466403 11962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:11.466423 11962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2061429195ca44b68f0dc26b2dc82deb, State: Initialized, Role: FOLLOWER
I20240503 08:14:11.466522 11962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2061429195ca44b68f0dc26b2dc82deb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36429 } }
I20240503 08:14:11.466590 11962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:14:11.466611 11962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:14:11.466660 11962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:11.467348 11962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2061429195ca44b68f0dc26b2dc82deb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36429 } }
I20240503 08:14:11.467444 11962 leader_election.cc:302] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2061429195ca44b68f0dc26b2dc82deb; no voters:
I20240503 08:14:11.467648 11962 leader_election.cc:288] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:14:11.467707 11967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:11.467866 11962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:11.467870 11967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [term 1 LEADER]: Becoming Leader. State: Replica: 2061429195ca44b68f0dc26b2dc82deb, State: Running, Role: LEADER
I20240503 08:14:11.467985 11967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2061429195ca44b68f0dc26b2dc82deb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36429 } }
I20240503 08:14:11.468304 11968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2061429195ca44b68f0dc26b2dc82deb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2061429195ca44b68f0dc26b2dc82deb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36429 } } }
I20240503 08:14:11.468326 11969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2061429195ca44b68f0dc26b2dc82deb. Latest consensus state: current_term: 1 leader_uuid: "2061429195ca44b68f0dc26b2dc82deb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2061429195ca44b68f0dc26b2dc82deb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36429 } } }
I20240503 08:14:11.468487 11968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:11.468508 11969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:11.468894 11976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:14:11.469391 11976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:14:11.470765 11976 catalog_manager.cc:1347] Generated new cluster ID: a89039996d8d4994819da4b884333fdd
I20240503 08:14:11.470822 11976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:14:11.480350 11976 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:14:11.480850 11976 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:14:11.496757 11976 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2061429195ca44b68f0dc26b2dc82deb: Generated new TSK 0
I20240503 08:14:11.496971 11976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:14:11.558308 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:11.507292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.6.68.254:36429)
0503 08:14:11.507347 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:11.507349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:11.507432 (+ 83us) server_negotiation.cc:407] Connection header received
0503 08:14:11.507489 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:11.507490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:11.507502 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:11.507600 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0503 08:14:11.507887 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.508681 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:11.509269 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.509452 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:11.509509 (+ 57us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0503 08:14:11.557982 (+ 48473us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0503 08:14:11.558200 (+ 218us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33688: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":30}
W20240503 08:14:11.558337 11991 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:11.507204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:36429 (local address 127.0.0.1:33688)
0503 08:14:11.507353 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:11.507363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:11.507427 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:11.507597 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:11.507598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:11.507599 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0503 08:14:11.507813 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:11.507816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.508787 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:11.508787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:11.509194 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:11.509195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:11.509253 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:11.509257 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0503 08:14:11.558126 (+ 48869us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0503 08:14:11.558257 (+ 131us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.68.254:36429: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20240503 08:14:11.558395 11965 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33688: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20240503 08:14:11.558712 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 11895
2024-05-03T08:14:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (237 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:14:11 dist-test-slave-3g67 krb5kdc[12003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:14:11 dist-test-slave-3g67 krb5kdc[12003](info): set up 2 sockets
May 03 08:14:11 dist-test-slave-3g67 krb5kdc[12003](info): commencing operation
krb5kdc: starting...
W20240503 08:14:12.696218 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:14:12 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:14:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:14:12.815337 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:35711
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:12.897821 12016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:12.898051 12016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:12.898088 12016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:12.899448 12016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:12.899508 12016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:12.899534 12016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:12.899557 12016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:12.899580 12016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:14:12.901357 12016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:35711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:12.901633 12016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:12.901840 12016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:12.906634 12024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:12.906950 12022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:12.907981 12021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:12.908674 12016 server_base.cc:1034] running on GCE node
I20240503 08:14:12.908911 12016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:12.909256 12016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:12.910436 12016 hybrid_clock.cc:648] HybridClock initialized: now 1714724052910412 us; error 39 us; skew 500 ppm
May 03 08:14:12 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724052, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:12.913048 12016 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:14:12.913489 12016 webserver.cc:460] Webserver started at http://127.6.68.254:32969/ using document root <none> and password file <none>
I20240503 08:14:12.913661 12016 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:12.913702 12016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:12.913796 12016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:12.914867 12016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "b82e9ccbcd3847a8a169a72109bb4637"
format_stamp: "Formatted at 2024-05-03 08:14:12 on dist-test-slave-3g67"
server_key: "ea8704834ce7f47692b697ceac8dc3d8"
server_key_iv: "ff6eda4ff685a756e0b4090a2eed3c03"
server_key_version: "encryptionkey@0"
I20240503 08:14:12.915169 12016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "b82e9ccbcd3847a8a169a72109bb4637"
format_stamp: "Formatted at 2024-05-03 08:14:12 on dist-test-slave-3g67"
server_key: "ea8704834ce7f47692b697ceac8dc3d8"
server_key_iv: "ff6eda4ff685a756e0b4090a2eed3c03"
server_key_version: "encryptionkey@0"
I20240503 08:14:12.916553 12016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:14:12.917270 12031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:12.917474 12016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:14:12.917562 12016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "b82e9ccbcd3847a8a169a72109bb4637"
format_stamp: "Formatted at 2024-05-03 08:14:12 on dist-test-slave-3g67"
server_key: "ea8704834ce7f47692b697ceac8dc3d8"
server_key_iv: "ff6eda4ff685a756e0b4090a2eed3c03"
server_key_version: "encryptionkey@0"
I20240503 08:14:12.917634 12016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:12.930910 12016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:12.932886 12016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:12.933007 12016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:12.936744 12016 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:35711
I20240503 08:14:12.936794 12083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:35711 every 8 connection(s)
I20240503 08:14:12.937096 12016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:14:12.937723 12084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:12.939810 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 12016
I20240503 08:14:12.939898 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:14:12.940052 6419 external_mini_cluster.cc:1428] Setting key c0ad2ea966cdde5cb89cbde486a7e9f2
I20240503 08:14:12.940933 12084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637: Bootstrap starting.
I20240503 08:14:12.941627 12084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:12.941946 12084 log.cc:824] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:12.942641 12084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637: No bootstrap required, opened a new log
I20240503 08:14:12.944743 12084 raft_consensus.cc:348] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82e9ccbcd3847a8a169a72109bb4637" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35711 } }
I20240503 08:14:12.944881 12084 raft_consensus.cc:374] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:12.944901 12084 raft_consensus.cc:729] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b82e9ccbcd3847a8a169a72109bb4637, State: Initialized, Role: FOLLOWER
I20240503 08:14:12.945001 12084 consensus_queue.cc:260] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82e9ccbcd3847a8a169a72109bb4637" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35711 } }
I20240503 08:14:12.945066 12084 raft_consensus.cc:388] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:14:12.945089 12084 raft_consensus.cc:482] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:14:12.945135 12084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:12.945709 12084 raft_consensus.cc:504] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82e9ccbcd3847a8a169a72109bb4637" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35711 } }
I20240503 08:14:12.945812 12084 leader_election.cc:302] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b82e9ccbcd3847a8a169a72109bb4637; no voters:
I20240503 08:14:12.945984 12084 leader_election.cc:288] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:14:12.946035 12089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:12.946194 12089 raft_consensus.cc:686] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [term 1 LEADER]: Becoming Leader. State: Replica: b82e9ccbcd3847a8a169a72109bb4637, State: Running, Role: LEADER
I20240503 08:14:12.946254 12084 sys_catalog.cc:564] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:12.946290 12089 consensus_queue.cc:237] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82e9ccbcd3847a8a169a72109bb4637" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35711 } }
May 03 08:14:12 dist-test-slave-3g67 krb5kdc[12003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:14:12.946583 12089 sys_catalog.cc:455] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b82e9ccbcd3847a8a169a72109bb4637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82e9ccbcd3847a8a169a72109bb4637" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35711 } } }
I20240503 08:14:12.946651 12089 sys_catalog.cc:458] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:12.946771 12089 sys_catalog.cc:455] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b82e9ccbcd3847a8a169a72109bb4637. Latest consensus state: current_term: 1 leader_uuid: "b82e9ccbcd3847a8a169a72109bb4637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b82e9ccbcd3847a8a169a72109bb4637" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35711 } } }
I20240503 08:14:12.946878 12089 sys_catalog.cc:458] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:12.947021 12094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:14:12.947527 12094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:14:12.950148 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:12.940583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.6.68.254:35711)
0503 08:14:12.940843 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:12.940855 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:12.940899 (+ 44us) server_negotiation.cc:407] Connection header received
0503 08:14:12.941527 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:12.941535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:12.941595 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:12.941817 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:12.942553 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:12.943783 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:12.944358 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:12.944648 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:12.946855 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:12.946866 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:12.946867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:12.946890 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:12.948858 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:12.949201 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:12.949205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:12.949206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:12.949256 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:12.949413 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:12.949413 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:12.949414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:12.949601 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:12.949755 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:12.949865 (+ 110us) server_negotiation.cc:299] Negotiation successful
0503 08:14:12.949996 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":74,"threads_started":1}
I20240503 08:14:12.952284 12094 catalog_manager.cc:1347] Generated new cluster ID: 01b69dd88659473b9ddc0fff7ef197df
I20240503 08:14:12.952350 12094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:14:12.964511 12094 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:14:12.965178 12094 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:14:12.979375 12094 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b82e9ccbcd3847a8a169a72109bb4637: Generated new TSK 0
I20240503 08:14:12.979591 12094 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:14:13.018887 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35711
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:13.104302 12110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:13.104507 12110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:13.104537 12110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:13.106003 12110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:13.106063 12110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:13.106093 12110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:14:13.107934 12110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:13.108170 12110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:13.108379 12110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:13.113386 12115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:13.113734 12118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:13.114647 12116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:13.115610 12110 server_base.cc:1034] running on GCE node
I20240503 08:14:13.115871 12110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:13.116184 12110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:13.117416 12110 hybrid_clock.cc:648] HybridClock initialized: now 1714724053117392 us; error 39 us; skew 500 ppm
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:13.120291 12110 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:14:13.120817 12110 webserver.cc:460] Webserver started at http://127.6.68.193:34059/ using document root <none> and password file <none>
I20240503 08:14:13.121016 12110 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:13.121064 12110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:13.121192 12110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:13.122304 12110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "d81b2af3f0e74a429f0af1d90876c24e"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-3g67"
server_key: "0c1408acdb91f6ef8e50614daf7742e6"
server_key_iv: "95c0402a6659d7951e58b354d810be18"
server_key_version: "encryptionkey@0"
I20240503 08:14:13.122628 12110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "d81b2af3f0e74a429f0af1d90876c24e"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-3g67"
server_key: "0c1408acdb91f6ef8e50614daf7742e6"
server_key_iv: "95c0402a6659d7951e58b354d810be18"
server_key_version: "encryptionkey@0"
I20240503 08:14:13.123952 12110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:14:13.124756 12125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:13.124943 12110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:14:13.125023 12110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "d81b2af3f0e74a429f0af1d90876c24e"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-3g67"
server_key: "0c1408acdb91f6ef8e50614daf7742e6"
server_key_iv: "95c0402a6659d7951e58b354d810be18"
server_key_version: "encryptionkey@0"
I20240503 08:14:13.125090 12110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:13.148864 12110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:13.151260 12110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:13.151407 12110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:13.151695 12110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:13.152019 12110 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:13.152053 12110 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:13.152089 12110 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:13.152124 12110 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:13.157563 12110 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:44941
I20240503 08:14:13.157649 12238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:44941 every 8 connection(s)
I20240503 08:14:13.157927 12110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:14:13.164525 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 12110
I20240503 08:14:13.164654 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:14:13.164822 6419 external_mini_cluster.cc:1428] Setting key 263e2286f1bbdcc5a47a4b67855d68cc
I20240503 08:14:13.167265 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:13.158662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:55669 (local address 127.6.68.254:35711)
0503 08:14:13.158711 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:13.158713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:13.159020 (+ 307us) server_negotiation.cc:407] Connection header received
0503 08:14:13.159510 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:13.159512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:13.159524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:13.159585 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:13.160422 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.160946 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:13.161495 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.161679 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:13.164112 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:13.164117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:13.164117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:13.164124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:13.165866 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.166295 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.166296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.166296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.166323 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.166540 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.166540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.166541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.166674 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:13.166735 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:13.167016 (+ 281us) server_negotiation.cc:299] Negotiation successful
0503 08:14:13.167131 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:14:13.167338 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:13.158616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:35711 (local address 127.6.68.193:55669)
0503 08:14:13.158867 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:13.158905 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:13.159396 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:13.159678 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:13.159683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:13.160004 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:13.160326 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:13.160331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.161032 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:13.161033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:13.161417 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:13.161418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.161812 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:13.162413 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:13.162432 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:13.163994 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:13.165955 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:13.165962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:13.165965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:13.166185 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:13.166409 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:13.166410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:13.166411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:13.166481 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:13.166729 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:13.166732 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:13.166847 (+ 115us) client_negotiation.cc:769] Sending connection context
0503 08:14:13.166982 (+ 135us) client_negotiation.cc:240] Negotiation successful
0503 08:14:13.167137 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20240503 08:14:13.167927 12239 heartbeater.cc:347] Connected to a master server at 127.6.68.254:35711
I20240503 08:14:13.168046 12239 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:13.168300 12239 heartbeater.cc:511] Master 127.6.68.254:35711 requested a full tablet report, sending...
I20240503 08:14:13.168900 12048 ts_manager.cc:194] Registered new tserver with Master: d81b2af3f0e74a429f0af1d90876c24e (127.6.68.193:44941)
I20240503 08:14:13.169598 12048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:55669
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:14:13.194763 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35711
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:13.277283 12244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:13.277493 12244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:13.277518 12244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:13.278913 12244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:13.278962 12244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:13.278990 12244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:14:13.280709 12244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:13.280937 12244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:13.281106 12244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:13.286331 12249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:13.286499 12250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:13.287122 12252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:13.288112 12244 server_base.cc:1034] running on GCE node
I20240503 08:14:13.288340 12244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:13.288566 12244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:13.289708 12244 hybrid_clock.cc:648] HybridClock initialized: now 1714724053289684 us; error 38 us; skew 500 ppm
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:13.292397 12244 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:14:13.292865 12244 webserver.cc:460] Webserver started at http://127.6.68.194:33055/ using document root <none> and password file <none>
I20240503 08:14:13.293045 12244 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:13.293089 12244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:13.293200 12244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:13.294214 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "655e4ab20db84de4b8e00a6332d0f8ec"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-3g67"
server_key: "d6cceee9c7c63590d34a3cea53370829"
server_key_iv: "463ecd47d1fef1cc12cccfa584ec1457"
server_key_version: "encryptionkey@0"
I20240503 08:14:13.294553 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "655e4ab20db84de4b8e00a6332d0f8ec"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-3g67"
server_key: "d6cceee9c7c63590d34a3cea53370829"
server_key_iv: "463ecd47d1fef1cc12cccfa584ec1457"
server_key_version: "encryptionkey@0"
I20240503 08:14:13.295904 12244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240503 08:14:13.296612 12259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:13.296777 12244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:14:13.296854 12244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "655e4ab20db84de4b8e00a6332d0f8ec"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-3g67"
server_key: "d6cceee9c7c63590d34a3cea53370829"
server_key_iv: "463ecd47d1fef1cc12cccfa584ec1457"
server_key_version: "encryptionkey@0"
I20240503 08:14:13.296919 12244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:13.308490 12244 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:13.310633 12244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:13.310758 12244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:13.310968 12244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:13.311268 12244 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:13.311301 12244 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:13.311336 12244 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:13.311370 12244 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:13.316895 12244 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:35329
I20240503 08:14:13.316955 12372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:35329 every 8 connection(s)
I20240503 08:14:13.317260 12244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:14:13.319556 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 12244
I20240503 08:14:13.319694 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:14:13.319890 6419 external_mini_cluster.cc:1428] Setting key fce6c4c3edec1fbaf96016c0791d2203
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:14:13.326961 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:13.318023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:56425 (local address 127.6.68.254:35711)
0503 08:14:13.318082 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:13.318085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:13.318247 (+ 162us) server_negotiation.cc:407] Connection header received
0503 08:14:13.318884 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:13.318886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:13.318899 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:13.318958 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:13.319981 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.320530 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:13.321254 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.321445 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:13.323947 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:13.323951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:13.323952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:13.323959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:13.325513 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.326042 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.326043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.326043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.326073 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.326324 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.326325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.326325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.326427 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:13.326483 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:13.326808 (+ 325us) server_negotiation.cc:299] Negotiation successful
0503 08:14:13.326888 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240503 08:14:13.327176 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:13.317989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:35711 (local address 127.6.68.194:56425)
0503 08:14:13.318138 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:13.318162 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:13.318740 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:13.319019 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:13.319022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:13.319384 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:13.319847 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:13.319852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.320612 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:13.320613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:13.321163 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:13.321166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.321496 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:13.322113 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:13.322126 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:13.323835 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:13.325607 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:13.325615 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:13.325619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:13.325940 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:13.326158 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:13.326158 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:13.326159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:13.326252 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:13.326495 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:13.326499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:13.326659 (+ 160us) client_negotiation.cc:769] Sending connection context
0503 08:14:13.326791 (+ 132us) client_negotiation.cc:240] Negotiation successful
0503 08:14:13.326952 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20240503 08:14:13.327805 12373 heartbeater.cc:347] Connected to a master server at 127.6.68.254:35711
I20240503 08:14:13.327927 12373 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:13.328195 12373 heartbeater.cc:511] Master 127.6.68.254:35711 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
I20240503 08:14:13.328634 12048 ts_manager.cc:194] Registered new tserver with Master: 655e4ab20db84de4b8e00a6332d0f8ec (127.6.68.194:35329)
I20240503 08:14:13.329107 12048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:56425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:14:13.347512 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35711
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:13.426942 12378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:13.427141 12378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:13.427165 12378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:13.428604 12378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:13.428659 12378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:13.428687 12378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:14:13.430305 12378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:13.430517 12378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:13.430723 12378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:13.435711 12384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:13.436182 12383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:13.436939 12386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:13.437772 12378 server_base.cc:1034] running on GCE node
I20240503 08:14:13.437974 12378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:13.438201 12378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:13.439347 12378 hybrid_clock.cc:648] HybridClock initialized: now 1714724053439337 us; error 27 us; skew 500 ppm
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:13.441902 12378 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:14:13.442294 12378 webserver.cc:460] Webserver started at http://127.6.68.195:41123/ using document root <none> and password file <none>
I20240503 08:14:13.442477 12378 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:13.442517 12378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:13.442617 12378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:13.443596 12378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "59854e4d58a24f1ca40962dbcc65f7e6"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-3g67"
server_key: "52f9f9484159c5cb79d76b979fcee2ac"
server_key_iv: "9dd315a446b98ea445bb7460feccce55"
server_key_version: "encryptionkey@0"
I20240503 08:14:13.443933 12378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "59854e4d58a24f1ca40962dbcc65f7e6"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-3g67"
server_key: "52f9f9484159c5cb79d76b979fcee2ac"
server_key_iv: "9dd315a446b98ea445bb7460feccce55"
server_key_version: "encryptionkey@0"
I20240503 08:14:13.445204 12378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:13.445942 12393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:13.446117 12378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:14:13.446192 12378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "59854e4d58a24f1ca40962dbcc65f7e6"
format_stamp: "Formatted at 2024-05-03 08:14:13 on dist-test-slave-3g67"
server_key: "52f9f9484159c5cb79d76b979fcee2ac"
server_key_iv: "9dd315a446b98ea445bb7460feccce55"
server_key_version: "encryptionkey@0"
I20240503 08:14:13.446265 12378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:13.461606 12378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:13.463318 12378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:13.463431 12378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:13.463699 12378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:13.464056 12378 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:13.464089 12378 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:13.464126 12378 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:13.464150 12378 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:13.469326 12378 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:34919
I20240503 08:14:13.469372 12506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:34919 every 8 connection(s)
I20240503 08:14:13.469681 12378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:14:13.472219 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 12378
I20240503 08:14:13.472313 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:14:13.472476 6419 external_mini_cluster.cc:1428] Setting key 78d3d3626b73efe153fd41bdb5e4c886
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:14:13.478047 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:13.470423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:52075 (local address 127.6.68.254:35711)
0503 08:14:13.470500 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:13.470503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:13.470679 (+ 176us) server_negotiation.cc:407] Connection header received
0503 08:14:13.471115 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:13.471116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:13.471126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:13.471182 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:13.471974 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.472687 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:13.473442 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.473555 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:13.475762 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:13.475765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:13.475766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:13.475772 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:13.476981 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.477330 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.477330 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.477331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.477345 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.477495 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.477495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.477495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.477579 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:13.477624 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:13.477899 (+ 275us) server_negotiation.cc:299] Negotiation successful
0503 08:14:13.477979 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240503 08:14:13.478075 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:13.470352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:35711 (local address 127.6.68.195:52075)
0503 08:14:13.470564 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:13.470593 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:13.471032 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:13.471236 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:13.471239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:13.471551 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:13.471896 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:13.471900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.472788 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:13.472790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:13.473363 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:13.473366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.473678 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:13.474231 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:13.474248 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:13.475676 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:13.477052 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:13.477057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:13.477059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:13.477250 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:13.477401 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:13.477401 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:13.477402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:13.477452 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:13.477622 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:13.477625 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:13.477730 (+ 105us) client_negotiation.cc:769] Sending connection context
0503 08:14:13.477818 (+ 88us) client_negotiation.cc:240] Negotiation successful
0503 08:14:13.477924 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":90,"threads_started":1}
I20240503 08:14:13.478585 12507 heartbeater.cc:347] Connected to a master server at 127.6.68.254:35711
I20240503 08:14:13.478683 12507 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:13.478888 12507 heartbeater.cc:511] Master 127.6.68.254:35711 requested a full tablet report, sending...
I20240503 08:14:13.479332 12048 ts_manager.cc:194] Registered new tserver with Master: 59854e4d58a24f1ca40962dbcc65f7e6 (127.6.68.195:34919)
I20240503 08:14:13.479817 12048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:52075
I20240503 08:14:13.484426 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/test-admin.keytab.
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:13.500931 6419 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20240503 08:14:13.502485 12514 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:13.501794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.6.68.193:44941)
0503 08:14:13.501951 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:13.501957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:13.501967 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:14:13.501995 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:13.501996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:13.502011 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:13.502084 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:13.502382 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43112: BlockingRecv error: recv got EOF from 127.0.0.1:43112 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
W20240503 08:14:13.502646 6419 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/krb5cc' of type FILE
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:13.503602 6419 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:13.504535 6419 init.cc:295] Successfully reacquired a new kerberos TGT
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.193@KRBTEST.COM
I20240503 08:14:13.511758 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:13.505132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.6.68.193:44941)
0503 08:14:13.505188 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:13.505190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:13.505215 (+ 25us) server_negotiation.cc:407] Connection header received
0503 08:14:13.505284 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:13.505285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:13.505296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:13.505344 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:13.505847 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.506564 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:13.507104 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.507265 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:13.509171 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:13.509182 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:13.509182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:13.509188 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:13.510714 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.511034 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.511037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.511037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.511067 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.511207 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.511207 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.511208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.511309 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:13.511457 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:13.511528 (+ 71us) server_negotiation.cc:299] Negotiation successful
0503 08:14:13.511646 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20240503 08:14:13.513521 12514 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:13.513027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.6.68.193:44941)
0503 08:14:13.513081 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:13.513083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:13.513093 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:14:13.513162 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:13.513163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:13.513171 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:13.513214 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:13.513483 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43126: BlockingRecv error: recv got EOF from 127.0.0.1:43126 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20240503 08:14:13.513633 6419 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/krb5cc' of type FILE
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:13.514571 6419 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:13.515368 6419 init.cc:295] Successfully reacquired a new kerberos TGT
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.193@KRBTEST.COM
I20240503 08:14:13.522099 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:13.515982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.6.68.193:44941)
0503 08:14:13.516055 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:13.516057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:13.516083 (+ 26us) server_negotiation.cc:407] Connection header received
0503 08:14:13.516160 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:13.516160 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:13.516169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:13.516220 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:13.516689 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.517171 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:13.517690 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.517884 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:13.519684 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:13.519687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:13.519688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:13.519694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:13.520999 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.521285 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.521286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.521286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.521313 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.521461 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.521461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.521462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.521598 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:13.521732 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:13.521889 (+ 157us) server_negotiation.cc:299] Negotiation successful
0503 08:14:13.521982 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20240503 08:14:13.523777 12514 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:13.523192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.6.68.193:44941)
0503 08:14:13.523249 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:13.523251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:13.523318 (+ 67us) server_negotiation.cc:407] Connection header received
0503 08:14:13.523389 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:13.523389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:13.523398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:13.523452 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:13.523723 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43134: BlockingRecv error: recv got EOF from 127.0.0.1:43134 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20240503 08:14:13.523870 6419 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714724013808011-6419-0/krb5kdc/krb5cc' of type FILE
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:13.524837 6419 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:13.525599 6419 init.cc:295] Successfully reacquired a new kerberos TGT
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.193@KRBTEST.COM
I20240503 08:14:13.531994 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:13.526136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.6.68.193:44941)
0503 08:14:13.526212 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:13.526214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:13.526266 (+ 52us) server_negotiation.cc:407] Connection header received
0503 08:14:13.526334 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:13.526335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:13.526344 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:13.526397 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:13.526870 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.527347 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:13.527987 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:13.528171 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:13.529910 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:13.529915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:13.529916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:13.529922 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:13.531068 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.531344 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.531345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.531346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.531368 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:13.531492 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:13.531492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:13.531493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:13.531566 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:13.531686 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:13.531813 (+ 127us) server_negotiation.cc:299] Negotiation successful
0503 08:14:13.531898 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240503 08:14:13.532639 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 12110
I20240503 08:14:13.537631 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 12244
I20240503 08:14:13.542627 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 12378
I20240503 08:14:13.547206 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 12016
2024-05-03T08:14:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1812 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12527](info): set up 2 sockets
May 03 08:14:13 dist-test-slave-3g67 krb5kdc[12527](info): commencing operation
krb5kdc: starting...
W20240503 08:14:14.508385 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:14:14 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:14:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:14:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.254.keytab.
Entry for principal oryx/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.254.keytab.
I20240503 08:14:14.643168 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:37387
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.254.keytab
--principal=oryx/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:37387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:14.723006 12540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:14.723204 12540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:14.723240 12540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:14.724562 12540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:14.724606 12540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:14.724627 12540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:14.724646 12540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:14.724663 12540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:14:14.726219 12540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:37387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.254.keytab
--principal=oryx/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:37387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:14.726440 12540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:14.726624 12540 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:14.731160 12546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:14.731638 12545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:14.732493 12548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:14.733243 12540 server_base.cc:1034] running on GCE node
I20240503 08:14:14.733598 12540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:14.733876 12540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:14.735030 12540 hybrid_clock.cc:648] HybridClock initialized: now 1714724054735001 us; error 46 us; skew 500 ppm
May 03 08:14:14 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724054, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:14.737603 12540 init.cc:377] Logged in from keytab as oryx/127.6.68.254@KRBTEST.COM (short username oryx)
I20240503 08:14:14.738036 12540 webserver.cc:460] Webserver started at http://127.6.68.254:42255/ using document root <none> and password file <none>
I20240503 08:14:14.738224 12540 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:14.738267 12540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:14.738375 12540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:14.739375 12540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "39d8111f3ddf436a859b99d7496b0ea0"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-3g67"
server_key: "3d4ae378037304219ad5262d9f41182d"
server_key_iv: "a04b1282c158fa816e5571e87bf28db6"
server_key_version: "encryptionkey@0"
I20240503 08:14:14.739729 12540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "39d8111f3ddf436a859b99d7496b0ea0"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-3g67"
server_key: "3d4ae378037304219ad5262d9f41182d"
server_key_iv: "a04b1282c158fa816e5571e87bf28db6"
server_key_version: "encryptionkey@0"
I20240503 08:14:14.740983 12540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:14.741708 12555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:14.741869 12540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:14:14.741950 12540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "39d8111f3ddf436a859b99d7496b0ea0"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-3g67"
server_key: "3d4ae378037304219ad5262d9f41182d"
server_key_iv: "a04b1282c158fa816e5571e87bf28db6"
server_key_version: "encryptionkey@0"
I20240503 08:14:14.742012 12540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:14.753010 12540 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:14.755015 12540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:14.755156 12540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:14.759052 12540 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:37387
I20240503 08:14:14.759079 12607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:37387 every 8 connection(s)
I20240503 08:14:14.759421 12540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:14:14.760255 12608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:14.762377 12608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0: Bootstrap starting.
I20240503 08:14:14.762929 12608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:14.763182 12608 log.cc:824] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:14.763859 12608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0: No bootstrap required, opened a new log
I20240503 08:14:14.765244 12608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } }
I20240503 08:14:14.765367 12608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:14.765385 12608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d8111f3ddf436a859b99d7496b0ea0, State: Initialized, Role: FOLLOWER
I20240503 08:14:14.765501 12608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } }
I20240503 08:14:14.765570 12608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:14:14.765594 12608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:14:14.765623 12608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:14.766167 12608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } }
I20240503 08:14:14.766252 12608 leader_election.cc:302] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39d8111f3ddf436a859b99d7496b0ea0; no voters:
I20240503 08:14:14.766407 12608 leader_election.cc:288] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:14:14.766619 12611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:14.766676 12608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:14.766752 12611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 39d8111f3ddf436a859b99d7496b0ea0, State: Running, Role: LEADER
I20240503 08:14:14.766906 12611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } }
I20240503 08:14:14.767341 12612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } }
I20240503 08:14:14.767341 12613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39d8111f3ddf436a859b99d7496b0ea0. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } }
I20240503 08:14:14.767439 12612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:14.767459 12613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:14.767679 12620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:14:14.767784 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 12540
I20240503 08:14:14.767881 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:14:14.768049 6419 external_mini_cluster.cc:1428] Setting key 1760c95229592e0bb0ff0c07b56b3207
I20240503 08:14:14.768182 12620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:14:14.769990 12620 catalog_manager.cc:1347] Generated new cluster ID: 55e5ba691b05443fa7fb548e5e38aebf
I20240503 08:14:14.770045 12620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 03 08:14:14 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:14.777426 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:14.768709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.6.68.254:37387)
0503 08:14:14.768993 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:14.769004 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:14.769027 (+ 23us) server_negotiation.cc:407] Connection header received
0503 08:14:14.769126 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:14.769132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:14.769178 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:14.769332 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:14.770268 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:14.771010 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:14.771594 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:14.771867 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:14.773966 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:14.773972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:14.773973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:14.773988 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:14.775738 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:14.776050 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:14.776053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:14.776053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:14.776245 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:14.776567 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:14.776568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:14.776568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:14.776785 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:14.776919 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:14.777069 (+ 150us) server_negotiation.cc:299] Negotiation successful
0503 08:14:14.777206 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":91,"threads_started":1}
I20240503 08:14:14.779453 12620 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:14:14.780002 12620 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:14:14.784499 12620 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0: Generated new TSK 0
I20240503 08:14:14.784673 12620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab.
Entry for principal oryx/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab.
I20240503 08:14:14.818951 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37387
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab
--principal=oryx/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:14.933487 12634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:14.933770 12634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:14.933808 12634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:14.936105 12634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:14.936182 12634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:14.936225 12634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:14:14.938819 12634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab
--principal=oryx/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:14.939085 12634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:14.939327 12634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:14.944129 12640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:14.944572 12639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:14.945433 12642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:14.946307 12634 server_base.cc:1034] running on GCE node
I20240503 08:14:14.946528 12634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:14.946753 12634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:14.947902 12634 hybrid_clock.cc:648] HybridClock initialized: now 1714724054947892 us; error 29 us; skew 500 ppm
May 03 08:14:14 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724054, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:14.950531 12634 init.cc:377] Logged in from keytab as oryx/127.6.68.193@KRBTEST.COM (short username oryx)
I20240503 08:14:14.950960 12634 webserver.cc:460] Webserver started at http://127.6.68.193:40709/ using document root <none> and password file <none>
I20240503 08:14:14.951141 12634 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:14.951187 12634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:14.951308 12634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:14.952353 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "1d2cb9aab096451d93597c0e6c573ba2"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-3g67"
server_key: "0f7081de1135f7b308a76d9e514579a2"
server_key_iv: "c7e714e4c9526b6a2924e162dd7d588f"
server_key_version: "encryptionkey@0"
I20240503 08:14:14.952654 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "1d2cb9aab096451d93597c0e6c573ba2"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-3g67"
server_key: "0f7081de1135f7b308a76d9e514579a2"
server_key_iv: "c7e714e4c9526b6a2924e162dd7d588f"
server_key_version: "encryptionkey@0"
I20240503 08:14:14.953888 12634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:14:14.954586 12649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:14.954766 12634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:14:14.954842 12634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "1d2cb9aab096451d93597c0e6c573ba2"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-3g67"
server_key: "0f7081de1135f7b308a76d9e514579a2"
server_key_iv: "c7e714e4c9526b6a2924e162dd7d588f"
server_key_version: "encryptionkey@0"
I20240503 08:14:14.954905 12634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:14.986835 12634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:14.988961 12634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:14.989094 12634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:14.989320 12634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:14.989630 12634 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:14.989666 12634 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:14.989711 12634 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:14.989742 12634 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:14.995385 12634 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:42003
I20240503 08:14:14.995429 12762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:42003 every 8 connection(s)
I20240503 08:14:14.995775 12634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
May 03 08:14:14 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724054, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:15.004101 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:14.996483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:49951 (local address 127.6.68.254:37387)
0503 08:14:14.996578 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:14.996580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:14.996671 (+ 91us) server_negotiation.cc:407] Connection header received
0503 08:14:14.997160 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:14.997161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:14.997174 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:14.997277 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:14.998353 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:14.998859 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:14.999390 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:14.999514 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.001473 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.001478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.001479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.001485 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.002936 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.003291 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.003292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.003292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.003313 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.003466 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.003467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.003467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.003560 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.003608 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.003938 (+ 330us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.004022 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20240503 08:14:15.004215 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:14.996422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:49951)
0503 08:14:14.996564 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:14.996589 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:14.997054 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:14.997340 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:14.997344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:14.997649 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:14.998246 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:14.998250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:14.998937 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:14.998938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:14.999315 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:14.999317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:14.999425 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:14.999996 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.000009 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.001372 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.003012 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.003017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.003020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.003220 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.003369 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.003369 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.003370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.003424 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.003677 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.003684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.003809 (+ 125us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.003924 (+ 115us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.004039 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20240503 08:14:15.004660 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 12634
I20240503 08:14:15.004770 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:14:15.004878 12763 heartbeater.cc:347] Connected to a master server at 127.6.68.254:37387
I20240503 08:14:15.004956 6419 external_mini_cluster.cc:1428] Setting key 255aabf43b1fdd99228d47b47b6f5388
I20240503 08:14:15.004995 12763 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:15.005309 12763 heartbeater.cc:511] Master 127.6.68.254:37387 requested a full tablet report, sending...
I20240503 08:14:15.005825 12572 ts_manager.cc:194] Registered new tserver with Master: 1d2cb9aab096451d93597c0e6c573ba2 (127.6.68.193:42003)
I20240503 08:14:15.006549 12572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:49951
WARNING: no policy specified for oryx/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab.
Entry for principal oryx/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab.
I20240503 08:14:15.030340 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37387
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab
--principal=oryx/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:15.111279 12768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:15.111508 12768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:15.111532 12768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:15.112829 12768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:15.112876 12768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:15.112901 12768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:14:15.114526 12768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab
--principal=oryx/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:15.114725 12768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:15.114885 12768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:15.119783 12773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.120319 12774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.120926 12776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:15.121857 12768 server_base.cc:1034] running on GCE node
I20240503 08:14:15.122072 12768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:15.122368 12768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:15.123524 12768 hybrid_clock.cc:648] HybridClock initialized: now 1714724055123512 us; error 38 us; skew 500 ppm
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:15.126165 12768 init.cc:377] Logged in from keytab as oryx/127.6.68.194@KRBTEST.COM (short username oryx)
I20240503 08:14:15.126601 12768 webserver.cc:460] Webserver started at http://127.6.68.194:33429/ using document root <none> and password file <none>
I20240503 08:14:15.126772 12768 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:15.126808 12768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:15.126909 12768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:15.127918 12768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "a91e5b34f54e4586a9eb8c9642ff1108"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "b76257b4b5fadc7aa575315e6f0942b6"
server_key_iv: "0ccbc9dc61a150cb8b8b9add13543616"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.128247 12768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "a91e5b34f54e4586a9eb8c9642ff1108"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "b76257b4b5fadc7aa575315e6f0942b6"
server_key_iv: "0ccbc9dc61a150cb8b8b9add13543616"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.129498 12768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:14:15.130198 12783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.130432 12768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:15.130515 12768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "a91e5b34f54e4586a9eb8c9642ff1108"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "b76257b4b5fadc7aa575315e6f0942b6"
server_key_iv: "0ccbc9dc61a150cb8b8b9add13543616"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.130568 12768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:15.148566 12768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:15.150790 12768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:15.150902 12768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:15.151087 12768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:15.151422 12768 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:15.151463 12768 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.151489 12768 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:15.151501 12768 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.157228 12768 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:40553
I20240503 08:14:15.157277 12896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:40553 every 8 connection(s)
I20240503 08:14:15.157608 12768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:15.165225 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 12768
I20240503 08:14:15.165328 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:14:15.165498 6419 external_mini_cluster.cc:1428] Setting key 9d487d9e9fd0f6508f5f1b744523689c
I20240503 08:14:15.166554 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.158449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:46809 (local address 127.6.68.254:37387)
0503 08:14:15.158519 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.158521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.158629 (+ 108us) server_negotiation.cc:407] Connection header received
0503 08:14:15.159112 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.159113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.159125 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.159190 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.160323 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.160851 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.161424 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.161553 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.163613 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.163648 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.163648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.163659 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.165140 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.165581 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.165583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.165584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.165616 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.165887 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.165888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.165888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.165994 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.166058 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.166362 (+ 304us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.166463 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240503 08:14:15.166919 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.158371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:46809)
0503 08:14:15.158509 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.158542 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.158996 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.159250 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.159253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.159592 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.160182 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.160188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.160955 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.160956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.161336 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.161338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.161448 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.161979 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.161992 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.163490 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.165233 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.165240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.165243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.165500 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.165702 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.165703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.165703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.165808 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.166056 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.166063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.166223 (+ 160us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.166344 (+ 121us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.166499 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20240503 08:14:15.167479 12897 heartbeater.cc:347] Connected to a master server at 127.6.68.254:37387
I20240503 08:14:15.167594 12897 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:15.167932 12897 heartbeater.cc:511] Master 127.6.68.254:37387 requested a full tablet report, sending...
I20240503 08:14:15.168439 12572 ts_manager.cc:194] Registered new tserver with Master: a91e5b34f54e4586a9eb8c9642ff1108 (127.6.68.194:40553)
I20240503 08:14:15.168833 12572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:46809
WARNING: no policy specified for oryx/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab.
Entry for principal oryx/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab.
I20240503 08:14:15.196480 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37387
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab
--principal=oryx/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:15.279779 12902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:15.279994 12902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:15.280028 12902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:15.281428 12902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:15.281494 12902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:15.281536 12902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:14:15.283210 12902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab
--principal=oryx/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:15.283461 12902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:15.283705 12902 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:15.288583 12910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.289039 12907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.289772 12908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:15.290659 12902 server_base.cc:1034] running on GCE node
I20240503 08:14:15.290858 12902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:15.291066 12902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:15.292222 12902 hybrid_clock.cc:648] HybridClock initialized: now 1714724055292212 us; error 37 us; skew 500 ppm
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:15.294991 12902 init.cc:377] Logged in from keytab as oryx/127.6.68.195@KRBTEST.COM (short username oryx)
I20240503 08:14:15.295418 12902 webserver.cc:460] Webserver started at http://127.6.68.195:36355/ using document root <none> and password file <none>
I20240503 08:14:15.295610 12902 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:15.295699 12902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:15.295817 12902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:15.296828 12902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "7c9c1dbb33e3e22eef9dca2acb8f4621"
server_key_iv: "3555b7f7d049fbbd7ccb933b963ae4ff"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.297142 12902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "7c9c1dbb33e3e22eef9dca2acb8f4621"
server_key_iv: "3555b7f7d049fbbd7ccb933b963ae4ff"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.298427 12902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:14:15.299156 12917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.299351 12902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:14:15.299429 12902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "7c9c1dbb33e3e22eef9dca2acb8f4621"
server_key_iv: "3555b7f7d049fbbd7ccb933b963ae4ff"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.299497 12902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:15.326890 12902 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:15.328933 12902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:15.329061 12902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:15.329315 12902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:15.329640 12902 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:15.329674 12902 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.329708 12902 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:15.329728 12902 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.335527 12902 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:44211
I20240503 08:14:15.335572 13030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:44211 every 8 connection(s)
I20240503 08:14:15.335899 12902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:14:15.341477 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 12902
I20240503 08:14:15.341570 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:14:15.341753 6419 external_mini_cluster.cc:1428] Setting key 56b6379119c9c804c5b7e000e1a56c0b
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:15.345381 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.336635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:52637 (local address 127.6.68.254:37387)
0503 08:14:15.336702 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.336704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.336859 (+ 155us) server_negotiation.cc:407] Connection header received
0503 08:14:15.337353 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.337354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.337367 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.337437 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.338670 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.339439 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.340025 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.340188 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.342438 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.342441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.342441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.342448 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.344145 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.344514 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.344515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.344515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.344534 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.344799 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.344800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.344800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.344908 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.344973 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.345224 (+ 251us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.345307 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240503 08:14:15.345517 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.336588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:52637)
0503 08:14:15.336734 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.336761 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.337247 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.337508 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.337512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.337839 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.338513 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.338519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.339516 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.339517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.339948 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.339950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.340097 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.340658 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.340673 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.342368 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.344232 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.344238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.344240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.344445 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.344609 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.344610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.344610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.344739 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.344977 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.344981 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.345099 (+ 118us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.345197 (+ 98us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.345345 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20240503 08:14:15.346083 13031 heartbeater.cc:347] Connected to a master server at 127.6.68.254:37387
I20240503 08:14:15.346179 13031 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:15.346387 13031 heartbeater.cc:511] Master 127.6.68.254:37387 requested a full tablet report, sending...
I20240503 08:14:15.346778 12572 ts_manager.cc:194] Registered new tserver with Master: 89d4ed61819c4fb88e70a6f87e3ff3a2 (127.6.68.195:44211)
I20240503 08:14:15.347127 12572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:52637
I20240503 08:14:15.353642 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.68.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.253.keytab.
Entry for principal oryx/127.6.68.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.253.keytab.
I20240503 08:14:15.380059 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.253:34149
--webserver_interface=127.6.68.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.253.keytab
--principal=oryx/127.6.68.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:37387,127.6.68.253:34149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:15.462194 13036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:15.462430 13036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:15.462463 13036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:15.463915 13036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:15.463972 13036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:15.463995 13036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:15.464015 13036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:15.464054 13036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:14:15.465869 13036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:37387,127.6.68.253:34149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.253.keytab
--principal=oryx/127.6.68.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.253:34149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.68.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
W20240503 08:14:15.466034 13036 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.68.254:37387,127.6.68.253:34149'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20240503 08:14:15.466199 13036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:15.466392 13036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:15.471207 13041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.471589 13044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.472590 13042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:15.473363 13036 server_base.cc:1034] running on GCE node
I20240503 08:14:15.473608 13036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:15.473865 13036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:15.475021 13036 hybrid_clock.cc:648] HybridClock initialized: now 1714724055475004 us; error 37 us; skew 500 ppm
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:15.477906 13036 init.cc:377] Logged in from keytab as oryx/127.6.68.253@KRBTEST.COM (short username oryx)
I20240503 08:14:15.478390 13036 webserver.cc:460] Webserver started at http://127.6.68.253:37023/ using document root <none> and password file <none>
I20240503 08:14:15.478578 13036 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:15.478626 13036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:15.478739 13036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:15.479833 13036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data/instance:
uuid: "7b64c951cd374095b0772627949c3b75"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "7a7fe07aafb9dfce5f08acafa772004c"
server_key_iv: "4798a3c2bb437296f79721745984d9af"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.480141 13036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/wal/instance:
uuid: "7b64c951cd374095b0772627949c3b75"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "7a7fe07aafb9dfce5f08acafa772004c"
server_key_iv: "4798a3c2bb437296f79721745984d9af"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.481482 13036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:14:15.482340 13051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.482570 13036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:15.482662 13036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/wal
uuid: "7b64c951cd374095b0772627949c3b75"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "7a7fe07aafb9dfce5f08acafa772004c"
server_key_iv: "4798a3c2bb437296f79721745984d9af"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.482740 13036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:15.499830 13036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:15.502146 13036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:15.502285 13036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:15.506328 13036 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.253:34149
I20240503 08:14:15.506361 13103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.253:34149 every 8 connection(s)
I20240503 08:14:15.506721 13036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
I20240503 08:14:15.507546 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.509012 13104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.253@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:15.514807 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 13036
I20240503 08:14:15.517717 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.509714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57114 (local address 127.6.68.254:37387)
0503 08:14:15.509818 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.509820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.509963 (+ 143us) server_negotiation.cc:407] Connection header received
0503 08:14:15.510457 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.510458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.510472 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.510537 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.511340 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.511878 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.512442 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.512608 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.514615 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.514622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.514622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.514630 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.516441 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.516825 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.516826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.516826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.516849 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.517035 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.517036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.517036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.517192 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.517257 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.517526 (+ 269us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.517619 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20240503 08:14:15.517880 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.509679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:57114)
0503 08:14:15.509854 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.509880 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.510342 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.510602 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.510605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.510934 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.511227 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.511230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.511973 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.511974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.512358 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.512359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.512500 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.513117 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.513134 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.514514 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.516519 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.516523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.516527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.516754 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.516904 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.516905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.516905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.516970 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.517255 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.517260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.517395 (+ 135us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.517499 (+ 104us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.517667 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20240503 08:14:15.518584 13104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.68.253@KRBTEST.COM
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.253@KRBTEST.COM for oryx/127.6.68.253@KRBTEST.COM
I20240503 08:14:15.523471 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.515381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.6.68.253:34149)
0503 08:14:15.515577 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.515592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.515612 (+ 20us) server_negotiation.cc:407] Connection header received
0503 08:14:15.515847 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.515852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.515870 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.515963 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.516716 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.517406 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.518160 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.518364 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.520646 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.520658 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.520659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.520664 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.522372 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.522697 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.522700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.522701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.522739 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.522919 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.522920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.522921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.523033 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.523166 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.523254 (+ 88us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.523340 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20240503 08:14:15.525781 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.518918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:34149 (local address 127.0.0.1:33170)
0503 08:14:15.518968 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.518977 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.519008 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.519296 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.519297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.519455 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.519615 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.519617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.520369 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.520370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.520736 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.520738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.520832 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.521454 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.521462 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.522616 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.524837 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.524838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.524838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.525067 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.525255 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.525256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.525256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.525282 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.525470 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.525471 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.525507 (+ 36us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.525564 (+ 57us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.525672 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20240503 08:14:15.525905 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.519008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.6.68.253:34149)
0503 08:14:15.519192 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.519193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.519201 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:14:15.519215 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.519216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.519223 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.519285 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.519719 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.520203 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.520831 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.521042 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.522699 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.522703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.522703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.522711 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.524748 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.525160 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.525161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.525162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.525188 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.525334 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.525335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.525335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.525417 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.525473 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.525588 (+ 115us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.525784 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":169,"thread_start_us":127,"threads_started":1}
I20240503 08:14:15.527243 13104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: Bootstrap starting.
I20240503 08:14:15.527858 13104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:15.528084 13104 log.cc:824] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:15.528892 13104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: No bootstrap required, opened a new log
I20240503 08:14:15.530265 13104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } }
I20240503 08:14:15.530381 13104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:15.530398 13104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b64c951cd374095b0772627949c3b75, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.530478 13104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } }
I20240503 08:14:15.530771 13104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:15.530747 13111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } } }
I20240503 08:14:15.530817 13111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 08:14:15.532619 13122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:14:15.532797 13122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:14:15.534518 13036 master_runner.cc:428] Detected that this master 7b64c951cd374095b0772627949c3b75 is joining an existing cluster
I20240503 08:14:15.534566 13036 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.68.253:34149
I20240503 08:14:15.542763 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.536082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.6.68.254:37387)
0503 08:14:15.536304 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.536307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.536315 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:14:15.536336 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.536336 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.536346 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.536401 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.537779 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.538574 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.539377 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.539561 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.540210 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.540214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.540214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.540221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.541783 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.542091 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.542092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.542092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.542117 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.542258 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.542258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.542259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.542355 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.542413 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.542565 (+ 152us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.542659 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20240503 08:14:15.542940 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.535859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:57116)
0503 08:14:15.535992 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.536003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.536065 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.536416 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.536417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.536584 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.536804 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.536805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.538675 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.538677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.539249 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.539251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.539351 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.539894 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.539898 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.540125 (+ 227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.541878 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.541879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.541880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.542023 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.542174 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.542174 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.542175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.542203 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.542408 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.542409 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.542457 (+ 48us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.542534 (+ 77us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.542655 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724054, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.254@KRBTEST.COM for oryx/127.6.68.253@KRBTEST.COM
I20240503 08:14:15.562924 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.545588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:34149 (local address 127.0.0.1:33186)
0503 08:14:15.545855 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.545882 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.545958 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.546114 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.546117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.546473 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.546651 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.546653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.547432 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.547433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.547846 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.547848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.547920 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.548546 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.548551 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.550169 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.561718 (+ 11549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.561731 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.561736 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.562002 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.562183 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.562183 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.562184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.562229 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.562489 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.562498 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.562574 (+ 76us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.562671 (+ 97us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.562803 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":106,"threads_started":1}
I20240503 08:14:15.562950 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.535904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:34149 (local address 127.0.0.1:33180)
0503 08:14:15.536039 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.536047 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.536089 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.536247 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.536248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.536464 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.536789 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.536791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.537765 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.537766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.538136 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.538138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.538226 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.538833 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.538838 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.539225 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.561981 (+ 22756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.561983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.561983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.562147 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.562314 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.562314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.562315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.562337 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.562564 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.562565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.562599 (+ 34us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.562674 (+ 75us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.562790 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20240503 08:14:15.562965 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.536038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.6.68.253:34149)
0503 08:14:15.536119 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.536121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.536130 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:14:15.536150 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.536151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.536173 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.536239 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.536886 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.537621 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.538281 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.538476 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.539316 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.539320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.539320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.539327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.561879 (+ 22552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.562233 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.562234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.562234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.562254 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.562390 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.562391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.562392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.562505 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.562572 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.562785 (+ 213us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.562853 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20240503 08:14:15.563134 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.545634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.6.68.253:34149)
0503 08:14:15.545684 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.545686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.545943 (+ 257us) server_negotiation.cc:407] Connection header received
0503 08:14:15.546041 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.546043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.546054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.546110 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.546733 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.547320 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.547921 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.548163 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.550282 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.550287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.550287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.550294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.561586 (+ 11292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.562087 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.562088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.562089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.562112 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.562286 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.562287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.562288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.562415 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.562484 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.562705 (+ 221us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.563022 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":237,"server-negotiator.queue_time_us":29}
I20240503 08:14:15.564105 12572 catalog_manager.cc:6968] Initiating ChangeConfig request to add master 127.6.68.253:34149: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7b64c951cd374095b0772627949c3b75"
member_type: NON_VOTER
last_known_addr {
host: "127.6.68.253"
port: 34149
}
attrs {
promote: true
}
}
dest_uuid: "39d8111f3ddf436a859b99d7496b0ea0"
cas_config_opid_index: -1
I20240503 08:14:15.564380 12572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } }
I20240503 08:14:15.564851 13132 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7b64c951cd374095b0772627949c3b75 (127.6.68.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } }
I20240503 08:14:15.565184 13078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:15.566084 13132 catalog_manager.cc:6972] Successfully completed master ChangeConfig request to add master 127.6.68.253:34149
I20240503 08:14:15.566280 13132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } }
I20240503 08:14:15.566469 13132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:15.566458 13036 master.cc:537] Master@127.6.68.253:34149 shutting down...
I20240503 08:14:15.566780 13078 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Refusing update from remote peer 39d8111f3ddf436a859b99d7496b0ea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:14:15.567137 13131 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:15.568361 12558 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20240503 08:14:15.568805 12558 consensus_peers.cc:485] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 -> Peer 7b64c951cd374095b0772627949c3b75 (127.6.68.253:34149): Couldn't send request to peer 7b64c951cd374095b0772627949c3b75. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:15.569433 13036 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:15.569516 13036 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:15.569568 13036 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: stopping tablet replica
I20240503 08:14:15.582350 13036 master.cc:552] Master@127.6.68.253:34149 shutdown complete.
I20240503 08:14:15.582413 13036 master_runner.cc:305] Clearing existing system tablet
I20240503 08:14:15.582579 13036 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:15.582629 13036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:15.583292 13036 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:14:15.583906 13136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.584012 13036 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:15.584059 13036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.584087 13036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/wal
uuid: "7b64c951cd374095b0772627949c3b75"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "7a7fe07aafb9dfce5f08acafa772004c"
server_key_iv: "4798a3c2bb437296f79721745984d9af"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.584862 13036 ts_tablet_manager.cc:1875] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:15.586467 13036 ts_tablet_manager.cc:1888] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20240503 08:14:15.586542 13036 log.cc:1196] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20240503 08:14:15.586767 13036 ts_tablet_manager.cc:1909] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: Deleting consensus metadata
I20240503 08:14:15.586917 13036 master_runner.cc:315] Copying system tablet from 127.6.68.254:37387
I20240503 08:14:15.587752 13036 tablet_copy_client.cc:308] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: tablet copy: Beginning tablet copy session from remote peer at address 127.6.68.254:37387
I20240503 08:14:15.594020 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.588072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:57126)
0503 08:14:15.588191 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.588203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.588296 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.588446 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.588447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.588658 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.588901 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.588904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.589636 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.589637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.590229 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.590231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.590319 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.590862 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.590867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.591231 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.593136 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.593138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.593138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.593314 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.593465 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.593466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.593466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.593488 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.593726 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.593727 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.593774 (+ 47us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.593834 (+ 60us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.593927 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20240503 08:14:15.594060 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.588181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.6.68.254:37387)
0503 08:14:15.588229 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.588231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.588276 (+ 45us) server_negotiation.cc:407] Connection header received
0503 08:14:15.588365 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.588365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.588375 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.588427 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.589033 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.589549 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.590337 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.590467 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.591351 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.591356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.591356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.591364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.593038 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.593384 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.593385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.593386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.593410 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.593534 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.593534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.593535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.593657 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.593724 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.593870 (+ 146us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.593959 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:14:15.594471 12592 tablet_copy_service.cc:140] P 39d8111f3ddf436a859b99d7496b0ea0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7b64c951cd374095b0772627949c3b75 ({username='oryx', principal='oryx/127.6.68.253@KRBTEST.COM'} at 127.0.0.1:57126)
I20240503 08:14:15.594575 12592 tablet_copy_service.cc:161] P 39d8111f3ddf436a859b99d7496b0ea0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7b64c951cd374095b0772627949c3b75 at {username='oryx', principal='oryx/127.6.68.253@KRBTEST.COM'} at 127.0.0.1:57126: session id = 7b64c951cd374095b0772627949c3b75-00000000000000000000000000000000
I20240503 08:14:15.595201 12592 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:15.595834 13036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20240503 08:14:15.596866 13036 tablet_copy_client.cc:783] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:15.596990 13036 tablet_copy_client.cc:647] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:15.597940 13036 tablet_copy_client.cc:523] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:15.599150 12592 tablet_copy_service.cc:342] P 39d8111f3ddf436a859b99d7496b0ea0: Request end of tablet copy session 7b64c951cd374095b0772627949c3b75-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.68.253@KRBTEST.COM'} at 127.0.0.1:57126
I20240503 08:14:15.599217 12592 tablet_copy_service.cc:433] P 39d8111f3ddf436a859b99d7496b0ea0: ending tablet copy session 7b64c951cd374095b0772627949c3b75-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7b64c951cd374095b0772627949c3b75
I20240503 08:14:15.602397 13036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:15.604446 13147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.605046 13146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.606079 13149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:15.606916 13036 server_base.cc:1034] running on GCE node
I20240503 08:14:15.607043 13036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:15.607211 13036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:15.608337 13036 hybrid_clock.cc:648] HybridClock initialized: now 1714724055608323 us; error 38 us; skew 500 ppm
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:15.609767 13036 init.cc:377] Logged in from keytab as oryx/127.6.68.253@KRBTEST.COM (short username oryx)
I20240503 08:14:15.610047 13036 webserver.cc:460] Webserver started at http://127.6.68.253:43847/ using document root <none> and password file <none>
I20240503 08:14:15.610188 13036 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:15.610222 13036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:15.610879 13036 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:14:15.611407 13156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.611550 13036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.611663 13036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/wal
uuid: "7b64c951cd374095b0772627949c3b75"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "7a7fe07aafb9dfce5f08acafa772004c"
server_key_iv: "4798a3c2bb437296f79721745984d9af"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.611733 13036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:15.622290 13036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:15.624174 13036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:15.627252 13036 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.253:34149
I20240503 08:14:15.627501 13208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.253:34149 every 8 connection(s)
I20240503 08:14:15.627655 13036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
I20240503 08:14:15.628428 13209 sys_catalog.cc:263] Verifying existing consensus state
I20240503 08:14:15.628803 13209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: Bootstrap starting.
I20240503 08:14:15.630962 13209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:15.631178 13209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: Bootstrap complete.
I20240503 08:14:15.631400 13209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } }
I20240503 08:14:15.631474 13209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7b64c951cd374095b0772627949c3b75, State: Initialized, Role: LEARNER
I20240503 08:14:15.631515 13209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } }
I20240503 08:14:15.631719 13209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:15.631760 13212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } }
I20240503 08:14:15.631848 13212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: This master's current role is: LEARNER
I20240503 08:14:15.633195 13223 catalog_manager.cc:1259] Loaded cluster ID: 55e5ba691b05443fa7fb548e5e38aebf
I20240503 08:14:15.633250 13223 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: loading cluster ID for follower catalog manager: success
I20240503 08:14:15.634212 13223 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: acquiring CA information for follower catalog manager: success
I20240503 08:14:15.634521 13223 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:14:15.682992 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.675940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.6.68.253:34149)
0503 08:14:15.676148 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.676150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.676162 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:14:15.676187 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.676188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.676197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.676262 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.677185 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.677924 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.678756 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.678906 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.680375 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.680380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.680380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.680387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.681962 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.682350 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.682350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.682351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.682371 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.682516 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.682517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.682517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.682611 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.682692 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.682815 (+ 123us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.682894 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for oryx/127.6.68.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.252.keytab.
Entry for principal oryx/127.6.68.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.252.keytab.
I20240503 08:14:15.712098 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.252:38567
--webserver_interface=127.6.68.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.252.keytab
--principal=oryx/127.6.68.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:37387,127.6.68.253:34149,127.6.68.252:38567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:15.795866 13227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:15.796082 13227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:15.796118 13227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:15.797505 13227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:15.797564 13227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:15.797576 13227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:15.797590 13227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:15.797601 13227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:14:15.799299 13227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:37387,127.6.68.253:34149,127.6.68.252:38567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.252.keytab
--principal=oryx/127.6.68.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.252:38567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.68.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:15.799515 13227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:15.799700 13227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:15.804126 13232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.804469 13235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.805251 13233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:15.806119 13227 server_base.cc:1034] running on GCE node
I20240503 08:14:15.806376 13227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:15.806618 13227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:15.807762 13227 hybrid_clock.cc:648] HybridClock initialized: now 1714724055807742 us; error 42 us; skew 500 ppm
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:15.810279 13227 init.cc:377] Logged in from keytab as oryx/127.6.68.252@KRBTEST.COM (short username oryx)
I20240503 08:14:15.810739 13227 webserver.cc:460] Webserver started at http://127.6.68.252:34033/ using document root <none> and password file <none>
I20240503 08:14:15.810899 13227 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:15.810935 13227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:15.811015 13227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:15.812105 13227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data/instance:
uuid: "7d6c48594ed342ed81acbbe20139375d"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "cfe0837cd64def47df4035be2206873c"
server_key_iv: "eba727bc5c34668112666c4ec97742dc"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.812386 13227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/wal/instance:
uuid: "7d6c48594ed342ed81acbbe20139375d"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "cfe0837cd64def47df4035be2206873c"
server_key_iv: "eba727bc5c34668112666c4ec97742dc"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.813624 13227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:14:15.814280 13242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.814446 13227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:14:15.814523 13227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/wal
uuid: "7d6c48594ed342ed81acbbe20139375d"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "cfe0837cd64def47df4035be2206873c"
server_key_iv: "eba727bc5c34668112666c4ec97742dc"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.814592 13227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:15.825013 13227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:15.826881 13227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:15.826994 13227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:15.830935 13227 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.252:38567
I20240503 08:14:15.830965 13294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.252:38567 every 8 connection(s)
I20240503 08:14:15.831290 13227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
I20240503 08:14:15.832192 13295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:15.833725 13295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:14:15.836874 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 13227
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.252@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.68.252@KRBTEST.COM
I20240503 08:14:15.842782 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.834509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.6.68.254:37387)
0503 08:14:15.834563 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.834565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.834628 (+ 63us) server_negotiation.cc:407] Connection header received
0503 08:14:15.835086 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.835087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.835096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.835157 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.835944 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.836432 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.837257 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.837461 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.840022 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.840028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.840028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.840035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.841549 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.841909 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.841911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.841911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.841934 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.842105 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.842106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.842106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.842197 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.842263 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.842594 (+ 331us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.842676 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240503 08:14:15.842914 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.834367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:57142)
0503 08:14:15.834530 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.834562 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.835002 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.835217 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.835221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.835516 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.835869 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.835873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.836549 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.836551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.837156 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.837159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.837303 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.837959 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.837984 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.839935 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.841624 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.841629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.841633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.841838 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.841985 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.841986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.841986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.842050 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.842274 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.842281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.842437 (+ 156us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.842579 (+ 142us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.842739 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20240503 08:14:15.843505 13295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:14:15.845599 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.837539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.6.68.252:38567)
0503 08:14:15.837730 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.837739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.837756 (+ 17us) server_negotiation.cc:407] Connection header received
0503 08:14:15.837777 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.837779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.837799 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.837875 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.838760 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.839751 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.840360 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.840537 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.842694 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.842703 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.842704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.842709 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.844526 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.844873 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.844875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.844875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.844901 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.845017 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.845018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.845018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.845105 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.845215 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.845316 (+ 101us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.845396 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.252@KRBTEST.COM for oryx/127.6.68.253@KRBTEST.COM
I20240503 08:14:15.849617 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.843867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.6.68.253:34149)
0503 08:14:15.843928 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.843930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.843939 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:14:15.843963 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.843964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.843980 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.844031 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.844401 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.845109 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.845862 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.846029 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.847470 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.847474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.847474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.847481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.848815 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.849082 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.849083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.849083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.849099 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.849199 (+ 100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.849200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.849200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.849279 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.849324 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.849486 (+ 162us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.849552 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:14:15.849617 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.843792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:34149 (local address 127.0.0.1:33230)
0503 08:14:15.843841 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.843851 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.843896 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.844048 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.844049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.844188 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.844327 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.844328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.845184 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.845185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.845789 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.845791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.845862 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.846290 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.846294 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.847389 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.848894 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.848895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.848895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.849022 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.849145 (+ 123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.849145 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.849146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.849168 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.849354 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.849355 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.849395 (+ 40us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.849470 (+ 75us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.849552 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20240503 08:14:15.850054 13295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.252" port: 38567 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.252@KRBTEST.COM for oryx/127.6.68.252@KRBTEST.COM
I20240503 08:14:15.855510 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.850300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:38567 (local address 127.0.0.1:52180)
0503 08:14:15.850344 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.850353 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.850394 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.850512 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.850513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.850644 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.850759 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.850761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.851345 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.851346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.851711 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.851712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.851775 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.852186 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.852190 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.853311 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.854735 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.854736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.854737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.854882 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.855006 (+ 124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.855007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.855007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.855029 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.855236 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.855237 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.855277 (+ 40us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.855343 (+ 66us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.855438 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20240503 08:14:15.855660 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.850343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.6.68.252:38567)
0503 08:14:15.850373 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.850375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.850382 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:14:15.850448 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.850449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.850459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.850510 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.850832 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.851262 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.851776 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.851952 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.853389 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.853392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.853392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.853397 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.854652 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.854945 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.854946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.854946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.854960 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.855070 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.855071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.855071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.855145 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.855190 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.855362 (+ 172us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.855499 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":17}
I20240503 08:14:15.857197 13295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: Bootstrap starting.
I20240503 08:14:15.857791 13295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:15.858033 13295 log.cc:824] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:15.858786 13295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: No bootstrap required, opened a new log
I20240503 08:14:15.860150 13295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 38567 } }
I20240503 08:14:15.860276 13295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:15.860293 13295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d6c48594ed342ed81acbbe20139375d, State: Initialized, Role: FOLLOWER
I20240503 08:14:15.860396 13295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 38567 } }
I20240503 08:14:15.860678 13295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:15.860670 13300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 38567 } } }
I20240503 08:14:15.860821 13300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: This master's current role is: FOLLOWER
W20240503 08:14:15.862252 13311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:14:15.862331 13311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:14:15.866518 13227 master_runner.cc:428] Detected that this master 7d6c48594ed342ed81acbbe20139375d is joining an existing cluster
I20240503 08:14:15.866570 13227 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.68.252:38567
I20240503 08:14:15.894520 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.867772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.6.68.253:34149)
0503 08:14:15.867807 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.867809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.868034 (+ 225us) server_negotiation.cc:407] Connection header received
0503 08:14:15.868152 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.868153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.868164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.868217 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.888505 (+ 20288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.889255 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.889954 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.890105 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.891526 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.891531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.891531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.891538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.893333 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.893749 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.893750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.893750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.893775 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.893953 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.893953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.893954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.894081 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.894123 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.894314 (+ 191us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.894412 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:14:15.894572 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.867974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.6.68.252:38567)
0503 08:14:15.868016 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.868018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.871843 (+ 3825us) server_negotiation.cc:407] Connection header received
0503 08:14:15.871914 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.871915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.871923 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.871974 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.888499 (+ 16525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.889169 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.889924 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.890080 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.891550 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.891552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.891552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.891558 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.893603 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.893912 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.893913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.893913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.893930 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.894061 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.894061 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.894062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.894153 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.894189 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.894367 (+ 178us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.894483 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240503 08:14:15.894832 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.867577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:34149 (local address 127.0.0.1:33244)
0503 08:14:15.867937 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.867948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.868074 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.868231 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.868232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.888051 (+ 19819us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.888372 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.888376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.889342 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.889343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.889871 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.889872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.890145 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.890868 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.890872 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.891416 (+ 544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.893434 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.893435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.893436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.893673 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.893853 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.893853 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.893854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.893886 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.894166 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.894167 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.894203 (+ 36us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.894608 (+ 405us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.894711 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":109,"threads_started":1}
I20240503 08:14:15.895386 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.867606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:57156)
0503 08:14:15.867971 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.867980 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.868075 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.868265 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.868266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.888024 (+ 19758us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.888340 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.888344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.889416 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.889417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.890034 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.890036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.890373 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.891026 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.891029 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.891507 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.894377 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.894378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.894378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.894599 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.894782 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.894783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.894783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.894813 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.895067 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.895068 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.895104 (+ 36us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.895173 (+ 69us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.895280 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":102,"threads_started":1}
I20240503 08:14:15.895567 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.867693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57156 (local address 127.6.68.254:37387)
0503 08:14:15.867779 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.867781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.868035 (+ 254us) server_negotiation.cc:407] Connection header received
0503 08:14:15.868150 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.868151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.868163 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.868192 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.888505 (+ 20313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.889306 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.890135 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.890317 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.891616 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.891646 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.891646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.891654 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.894276 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.894681 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.894682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.894683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.894707 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.894878 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.894878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.894879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.894986 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.895025 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.895380 (+ 355us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.895472 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20240503 08:14:15.896004 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.867805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:38567 (local address 127.0.0.1:52188)
0503 08:14:15.871749 (+ 3944us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.871762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.871851 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.871995 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.871996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.888050 (+ 16054us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.888386 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.888387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.889276 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.889277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.889833 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.889834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.890134 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.890807 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.890811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.891474 (+ 663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.893678 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.893679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.893679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.893844 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.893987 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.893988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.893988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.894008 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.894227 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.894227 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.894258 (+ 31us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.894408 (+ 150us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.895914 (+ 1506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3922,"thread_start_us":69,"threads_started":1}
I20240503 08:14:15.897100 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 12634
I20240503 08:14:15.904067 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:42003
--local_ip_for_outbound_sockets=127.6.68.193
--tserver_master_addrs=127.6.68.254:37387
--webserver_port=40709
--webserver_interface=127.6.68.193
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab
--principal=oryx/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.68.254:37387,127.6.68.253:34149
--tserver_master_addrs=127.6.68.254:37387,127.6.68.253:34149,127.6.68.252:38567 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724054, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.254@KRBTEST.COM for oryx/127.6.68.252@KRBTEST.COM
I20240503 08:14:15.910456 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.900615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:38567 (local address 127.0.0.1:52194)
0503 08:14:15.902926 (+ 2311us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.902940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.903004 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.903168 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.903170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.903368 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.903549 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.903552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.904368 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.904369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.904920 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.904922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.905000 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.905669 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.905676 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.907300 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.909414 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.909415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.909415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.909622 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.909805 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.909806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.909806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.909839 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.910101 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.910102 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.910156 (+ 54us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.910228 (+ 72us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.910343 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2286}
I20240503 08:14:15.910877 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.900667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.6.68.252:38567)
0503 08:14:15.900724 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.900726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.902994 (+ 2268us) server_negotiation.cc:407] Connection header received
0503 08:14:15.903082 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.903083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.903093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.903147 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.903644 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.904270 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.905011 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.905205 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.907396 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.907400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.907401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.907408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.909312 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.909704 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.909705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.909705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.909733 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.909901 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.909902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.909902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.910034 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.910110 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.910272 (+ 162us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.910407 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240503 08:14:15.911696 12572 catalog_manager.cc:6968] Initiating ChangeConfig request to add master 127.6.68.252:38567: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7d6c48594ed342ed81acbbe20139375d"
member_type: NON_VOTER
last_known_addr {
host: "127.6.68.252"
port: 38567
}
attrs {
promote: true
}
}
dest_uuid: "39d8111f3ddf436a859b99d7496b0ea0"
cas_config_opid_index: 5
I20240503 08:14:15.911965 12572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
I20240503 08:14:15.912487 13132 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7d6c48594ed342ed81acbbe20139375d (127.6.68.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } } }
I20240503 08:14:15.913465 13132 catalog_manager.cc:6972] Successfully completed master ChangeConfig request to add master 127.6.68.252:38567
I20240503 08:14:15.913535 13132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } } }
I20240503 08:14:15.913602 13132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:15.913951 13269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:15.915055 13269 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 1 FOLLOWER]: Refusing update from remote peer 39d8111f3ddf436a859b99d7496b0ea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:15.915325 13131 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:15.915779 13227 master.cc:537] Master@127.6.68.252:38567 shutting down...
I20240503 08:14:15.916651 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.912451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:34149 (local address 127.0.0.1:33254)
0503 08:14:15.913930 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.913940 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.913982 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.914264 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.914265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.914267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:15.914466 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.914467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.915415 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.915416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.916387 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.916390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.916462 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.916511 (+ 49us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.916546 (+ 35us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.916584 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463}
I20240503 08:14:15.917538 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.912527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.6.68.253:34149)
0503 08:14:15.912573 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.912574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.914036 (+ 1462us) server_negotiation.cc:407] Connection header received
0503 08:14:15.914067 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.914068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.914096 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.914180 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:15.914542 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.915321 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.916787 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.917296 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.917388 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.917412 (+ 24us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.917451 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:14:15.917791 13183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 LEARNER]: Refusing update from remote peer 39d8111f3ddf436a859b99d7496b0ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:14:15.918035 13131 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:14:15.918545 13183 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7d6c48594ed342ed81acbbe20139375d (127.6.68.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } } }
I20240503 08:14:15.918829 13269 raft_consensus.cc:858] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 38567 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
W20240503 08:14:15.918879 13269 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 38567 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
I20240503 08:14:15.919275 13269 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7b64c951cd374095b0772627949c3b75 (127.6.68.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } }
I20240503 08:14:15.919659 13212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39d8111f3ddf436a859b99d7496b0ea0. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } } }
I20240503 08:14:15.919734 13212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: This master's current role is: LEARNER
I20240503 08:14:15.919768 13212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } } }
I20240503 08:14:15.919803 13212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: This master's current role is: LEARNER
I20240503 08:14:15.920420 13300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39d8111f3ddf436a859b99d7496b0ea0. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } }
I20240503 08:14:15.920508 13300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20240503 08:14:15.920562 13300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } }
I20240503 08:14:15.920596 13300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20240503 08:14:15.920624 13300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: NON_VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: true } } }
I20240503 08:14:15.920653 13300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20240503 08:14:15.921057 13131 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0: attempting to promote NON_VOTER 7b64c951cd374095b0772627949c3b75 to VOTER
I20240503 08:14:15.921247 13131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
I20240503 08:14:15.921964 13183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 LEARNER]: Refusing update from remote peer 39d8111f3ddf436a859b99d7496b0ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:14:15.922482 13322 mvcc.cc:205] Tried to move back new op lower bound from 7023509728395595776 to 7023509728325529600. Current Snapshot: MvccSnapshot[applied={T|T < 7023509728334893056}]
I20240503 08:14:15.922480 13132 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:14:15.922646 13227 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
W20240503 08:14:15.922708 12558 consensus_peers.cc:485] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 -> Peer 7d6c48594ed342ed81acbbe20139375d (127.6.68.252:38567): Couldn't send request to peer 7d6c48594ed342ed81acbbe20139375d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:14:15.922719 13227 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20240503 08:14:15.922758 13227 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: stopping tablet replica
I20240503 08:14:15.923225 13132 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7b64c951cd374095b0772627949c3b75 (127.6.68.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } } }
I20240503 08:14:15.923398 13183 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7b64c951cd374095b0772627949c3b75 (127.6.68.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } } }
I20240503 08:14:15.924160 13132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } } }
I20240503 08:14:15.924222 13132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:15.924268 13132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } } }
I20240503 08:14:15.924304 13132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:15.927759 13325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39d8111f3ddf436a859b99d7496b0ea0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } } }
I20240503 08:14:15.927884 13325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:14:15.939105 13227 master.cc:552] Master@127.6.68.252:38567 shutdown complete.
I20240503 08:14:15.939172 13227 master_runner.cc:305] Clearing existing system tablet
I20240503 08:14:15.939307 13227 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:15.939360 13227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:15.940158 13227 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:15.940696 13332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.940856 13227 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:15.940932 13227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.940954 13227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/wal
uuid: "7d6c48594ed342ed81acbbe20139375d"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "cfe0837cd64def47df4035be2206873c"
server_key_iv: "eba727bc5c34668112666c4ec97742dc"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.941632 13227 ts_tablet_manager.cc:1875] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:15.943310 13227 ts_tablet_manager.cc:1888] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20240503 08:14:15.943383 13227 log.cc:1196] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20240503 08:14:15.943662 13227 ts_tablet_manager.cc:1909] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: Deleting consensus metadata
I20240503 08:14:15.943804 13227 master_runner.cc:315] Copying system tablet from 127.6.68.254:37387
I20240503 08:14:15.944648 13227 tablet_copy_client.cc:308] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.68.254:37387
I20240503 08:14:15.952263 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.944988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:57166)
0503 08:14:15.945156 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:15.945167 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:15.945259 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:15.945439 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:15.945440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:15.945656 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:15.945924 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.945927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.946870 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.946871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:15.947502 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:15.947504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.947583 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.948783 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:15.948788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:15.949171 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:15.951133 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.951135 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.951135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.951347 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.951537 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:15.951537 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:15.951538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:15.951568 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:15.951881 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:15.951882 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:15.951936 (+ 54us) client_negotiation.cc:769] Sending connection context
0503 08:14:15.952012 (+ 76us) client_negotiation.cc:240] Negotiation successful
0503 08:14:15.952128 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20240503 08:14:15.952297 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:15.945053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.6.68.254:37387)
0503 08:14:15.945108 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:15.945110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:15.945271 (+ 161us) server_negotiation.cc:407] Connection header received
0503 08:14:15.945312 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:15.945313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:15.945327 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:15.945397 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:15.946013 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.946762 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:15.947588 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:15.947838 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:15.949280 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:15.949286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:15.949286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:15.949295 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:15.951035 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.951432 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.951434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.951434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.951465 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:15.951655 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:15.951656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:15.951657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:15.951798 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:15.951891 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:15.952054 (+ 163us) server_negotiation.cc:299] Negotiation successful
0503 08:14:15.952160 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240503 08:14:15.953006 12592 tablet_copy_service.cc:140] P 39d8111f3ddf436a859b99d7496b0ea0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7d6c48594ed342ed81acbbe20139375d ({username='oryx', principal='oryx/127.6.68.252@KRBTEST.COM'} at 127.0.0.1:57166)
I20240503 08:14:15.953083 12592 tablet_copy_service.cc:161] P 39d8111f3ddf436a859b99d7496b0ea0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7d6c48594ed342ed81acbbe20139375d at {username='oryx', principal='oryx/127.6.68.252@KRBTEST.COM'} at 127.0.0.1:57166: session id = 7d6c48594ed342ed81acbbe20139375d-00000000000000000000000000000000
I20240503 08:14:15.953676 12592 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:14:15.954138 13227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20240503 08:14:15.955449 13227 tablet_copy_client.cc:783] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: tablet copy: Starting download of 0 data blocks...
I20240503 08:14:15.955586 13227 tablet_copy_client.cc:647] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:14:15.956326 13227 tablet_copy_client.cc:523] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:14:15.957253 12592 tablet_copy_service.cc:342] P 39d8111f3ddf436a859b99d7496b0ea0: Request end of tablet copy session 7d6c48594ed342ed81acbbe20139375d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.68.252@KRBTEST.COM'} at 127.0.0.1:57166
I20240503 08:14:15.957314 12592 tablet_copy_service.cc:433] P 39d8111f3ddf436a859b99d7496b0ea0: ending tablet copy session 7d6c48594ed342ed81acbbe20139375d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7d6c48594ed342ed81acbbe20139375d
I20240503 08:14:15.960188 13227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:15.962133 13342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.962738 13343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:15.963759 13345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:15.964530 13227 server_base.cc:1034] running on GCE node
I20240503 08:14:15.964706 13227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:15.964857 13227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:15.965975 13227 hybrid_clock.cc:648] HybridClock initialized: now 1714724055965966 us; error 30 us; skew 500 ppm
May 03 08:14:15 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724055, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:15.967772 13227 init.cc:377] Logged in from keytab as oryx/127.6.68.252@KRBTEST.COM (short username oryx)
I20240503 08:14:15.968061 13227 webserver.cc:460] Webserver started at http://127.6.68.252:41641/ using document root <none> and password file <none>
I20240503 08:14:15.968183 13227 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:15.968218 13227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:15.968976 13227 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:14:15.969883 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:15.970232 13227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:14:15.970345 13227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/wal
uuid: "7d6c48594ed342ed81acbbe20139375d"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "cfe0837cd64def47df4035be2206873c"
server_key_iv: "eba727bc5c34668112666c4ec97742dc"
server_key_version: "encryptionkey@0"
I20240503 08:14:15.970480 13227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240503 08:14:16.004276 13320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:16.004503 13320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:16.004539 13320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:16.005888 13320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:16.005936 13320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:16.005978 13320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:14:16.007697 13320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab
--principal=oryx/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:42003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=40709
--tserver_master_addrs=127.6.68.254:37387,127.6.68.253:34149,127.6.68.252:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:16.007937 13320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:16.008139 13320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:16.013736 13362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:16.014200 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:16.014837 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:16.015755 13320 server_base.cc:1034] running on GCE node
I20240503 08:14:16.015796 13227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:16.015976 13320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:16.016238 13320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:16.017397 13320 hybrid_clock.cc:648] HybridClock initialized: now 1714724056017363 us; error 39 us; skew 500 ppm
I20240503 08:14:16.017968 13227 kserver.cc:163] Server-wide thread pool size limit: 3276
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:16.020769 13320 init.cc:377] Logged in from keytab as oryx/127.6.68.193@KRBTEST.COM (short username oryx)
I20240503 08:14:16.021000 13227 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.252:38567
I20240503 08:14:16.021320 13320 webserver.cc:460] Webserver started at http://127.6.68.193:40709/ using document root <none> and password file <none>
I20240503 08:14:16.021347 13227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
I20240503 08:14:16.021561 13320 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:16.021629 13320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:16.022264 13418 sys_catalog.cc:263] Verifying existing consensus state
I20240503 08:14:16.022784 13418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: Bootstrap starting.
I20240503 08:14:16.022961 13320 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:16.023519 13421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.023703 13320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.023779 13320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "1d2cb9aab096451d93597c0e6c573ba2"
format_stamp: "Formatted at 2024-05-03 08:14:14 on dist-test-slave-3g67"
server_key: "0f7081de1135f7b308a76d9e514579a2"
server_key_iv: "c7e714e4c9526b6a2924e162dd7d588f"
server_key_version: "encryptionkey@0"
I20240503 08:14:16.024087 13320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:16.025122 13418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:16.025347 13418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: Bootstrap complete.
I20240503 08:14:16.025585 13418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
I20240503 08:14:16.025681 13418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d6c48594ed342ed81acbbe20139375d, State: Initialized, Role: LEARNER
I20240503 08:14:16.025728 13418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
I20240503 08:14:16.025909 13418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:16.027520 13434 catalog_manager.cc:1259] Loaded cluster ID: 55e5ba691b05443fa7fb548e5e38aebf
I20240503 08:14:16.027565 13434 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: loading cluster ID for follower catalog manager: success
I20240503 08:14:16.028745 13434 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: acquiring CA information for follower catalog manager: success
I20240503 08:14:16.029073 13434 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:14:16.029259 13423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } } }
I20240503 08:14:16.029348 13423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6c48594ed342ed81acbbe20139375d [sys.catalog]: This master's current role is: LEARNER
I20240503 08:14:16.031198 13416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.252:38567 every 8 connection(s)
I20240503 08:14:16.032578 13320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:16.034672 13320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:16.034793 13320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:16.034986 13320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:16.035264 13320 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:16.035295 13320 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.035347 13320 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:16.035375 13320 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.040766 13320 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:42003
I20240503 08:14:16.040818 13547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:42003 every 8 connection(s)
I20240503 08:14:16.041149 13320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.252@KRBTEST.COM
I20240503 08:14:16.049511 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 13320
I20240503 08:14:16.049679 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 12768
I20240503 08:14:16.051065 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.041876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:34647)
0503 08:14:16.042011 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.042041 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.042583 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.042817 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.042820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.043121 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:16.043417 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.043420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.044368 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.044369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.044944 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.044946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.045099 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.045754 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:16.045776 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:16.047490 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:16.049585 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.049592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.049595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.049880 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.050062 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.050063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.050063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.050145 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.050417 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:16.050422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:16.050573 (+ 151us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.050687 (+ 114us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.050836 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20240503 08:14:16.051334 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.042148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:58257 (local address 127.6.68.252:38567)
0503 08:14:16.042275 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.042277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.042359 (+ 82us) server_negotiation.cc:407] Connection header received
0503 08:14:16.042741 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.042742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.042754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.042810 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.044228 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.045369 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.046278 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.046481 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.048032 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.048058 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.048058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.048064 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.050203 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.050524 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.050525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.050525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.050548 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.050693 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.050694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.050694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.050789 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.050845 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.051121 (+ 276us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.051211 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20240503 08:14:16.051491 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.041885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:34647 (local address 127.6.68.254:37387)
0503 08:14:16.041935 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.041937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.042128 (+ 191us) server_negotiation.cc:407] Connection header received
0503 08:14:16.042684 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.042685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.042698 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.042758 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.043510 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.044293 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.045042 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.045217 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.047584 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.047590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.047590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.047597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.049493 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.049969 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.049970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.049971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.049996 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.050210 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.050211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.050212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.050333 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.050392 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.051315 (+ 923us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.051402 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:14:16.051700 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.041877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:38567 (local address 127.6.68.193:58257)
0503 08:14:16.042283 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.042293 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.042606 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.042890 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.042891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.043568 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:16.044140 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.044143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.045452 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.045453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.046028 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.046030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.046093 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.046576 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:16.046580 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:16.047933 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:16.050280 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.050280 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.050280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.050457 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.050607 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.050608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.050608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.050633 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.050869 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:16.050869 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:16.050901 (+ 32us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.050959 (+ 58us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.051037 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20240503 08:14:16.056835 13548 heartbeater.cc:347] Connected to a master server at 127.6.68.252:38567
I20240503 08:14:16.056947 13548 heartbeater.cc:464] Registering TS with master...
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.253@KRBTEST.COM
I20240503 08:14:16.057189 13548 heartbeater.cc:511] Master 127.6.68.252:38567 requested a full tablet report, sending...
I20240503 08:14:16.057569 13549 heartbeater.cc:347] Connected to a master server at 127.6.68.254:37387
I20240503 08:14:16.057627 13549 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:16.057732 13549 heartbeater.cc:511] Master 127.6.68.254:37387 requested a full tablet report, sending...
I20240503 08:14:16.058009 12572 ts_manager.cc:194] Re-registered known tserver with Master: 1d2cb9aab096451d93597c0e6c573ba2 (127.6.68.193:42003)
I20240503 08:14:16.058262 13377 ts_manager.cc:194] Registered new tserver with Master: 1d2cb9aab096451d93597c0e6c573ba2 (127.6.68.193:42003)
I20240503 08:14:16.058542 12572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:34647
I20240503 08:14:16.059078 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:40553
--local_ip_for_outbound_sockets=127.6.68.194
--tserver_master_addrs=127.6.68.254:37387
--webserver_port=33429
--webserver_interface=127.6.68.194
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab
--principal=oryx/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.68.254:37387,127.6.68.253:34149
--tserver_master_addrs=127.6.68.254:37387,127.6.68.253:34149,127.6.68.252:38567 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:14:16.060490 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.044414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:39957 (local address 127.6.68.253:34149)
0503 08:14:16.044471 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.044473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.044480 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:14:16.044502 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.044503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.044512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.044556 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.045747 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.046790 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.047569 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.047779 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.057487 (+ 9708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.057493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.057493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.057502 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.059379 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.059842 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.059843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.059843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.059866 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.060016 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.060016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.060017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.060112 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.060168 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.060302 (+ 134us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.060399 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240503 08:14:16.060508 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.043936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:34149 (local address 127.6.68.193:39957)
0503 08:14:16.044085 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.044095 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.044145 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.044607 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.044607 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.044764 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:16.044992 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.044994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.046901 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.046902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.047462 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.047465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.048440 (+ 975us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.049038 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:16.049042 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:16.057369 (+ 8327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:16.059499 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.059500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.059501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.059717 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.059933 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.059934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.059934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.059962 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.060165 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:16.060166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:16.060210 (+ 44us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.060272 (+ 62us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.060395 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20240503 08:14:16.060932 13550 heartbeater.cc:347] Connected to a master server at 127.6.68.253:34149
I20240503 08:14:16.061003 13550 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:16.061069 13550 heartbeater.cc:511] Master 127.6.68.253:34149 requested a full tablet report, sending...
I20240503 08:14:16.061445 13173 ts_manager.cc:194] Registered new tserver with Master: 1d2cb9aab096451d93597c0e6c573ba2 (127.6.68.193:42003)
W20240503 08:14:16.145036 13556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:16.145236 13556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:16.145262 13556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:16.146577 13556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:16.146620 13556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:16.146651 13556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:14:16.148252 13556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab
--principal=oryx/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:40553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=33429
--tserver_master_addrs=127.6.68.254:37387,127.6.68.253:34149,127.6.68.252:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:16.148444 13556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:16.148618 13556 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:16.153434 13561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:16.153898 13564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:16.154525 13562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:16.155472 13556 server_base.cc:1034] running on GCE node
I20240503 08:14:16.155695 13556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:16.155917 13556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:16.157068 13556 hybrid_clock.cc:648] HybridClock initialized: now 1714724056157049 us; error 38 us; skew 500 ppm
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:16.159611 13556 init.cc:377] Logged in from keytab as oryx/127.6.68.194@KRBTEST.COM (short username oryx)
I20240503 08:14:16.160043 13556 webserver.cc:460] Webserver started at http://127.6.68.194:33429/ using document root <none> and password file <none>
I20240503 08:14:16.160230 13556 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:16.160293 13556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:16.161615 13556 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:14:16.162155 13571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.162299 13556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.162382 13556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "a91e5b34f54e4586a9eb8c9642ff1108"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "b76257b4b5fadc7aa575315e6f0942b6"
server_key_iv: "0ccbc9dc61a150cb8b8b9add13543616"
server_key_version: "encryptionkey@0"
I20240503 08:14:16.162669 13556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:16.173018 13556 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:16.174772 13556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:16.174902 13556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:16.175105 13556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:16.175411 13556 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:16.175444 13556 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.175482 13556 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:16.175513 13556 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.181208 13556 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:40553
I20240503 08:14:16.181267 13684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:40553 every 8 connection(s)
I20240503 08:14:16.181596 13556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:14:16.185030 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 13556
I20240503 08:14:16.185225 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 12902
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.253@KRBTEST.COM
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.252@KRBTEST.COM
I20240503 08:14:16.191114 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:44211
--local_ip_for_outbound_sockets=127.6.68.195
--tserver_master_addrs=127.6.68.254:37387
--webserver_port=36355
--webserver_interface=127.6.68.195
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab
--principal=oryx/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.68.254:37387,127.6.68.253:34149
--tserver_master_addrs=127.6.68.254:37387,127.6.68.253:34149,127.6.68.252:38567 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:14:16.193995 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.182329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:45633 (local address 127.6.68.252:38567)
0503 08:14:16.182376 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.182379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.182622 (+ 243us) server_negotiation.cc:407] Connection header received
0503 08:14:16.183154 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.183156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.183173 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.183226 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.184292 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.185008 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.188981 (+ 3973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.189211 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.190948 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.190952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.190953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.190961 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.192865 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.193210 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.193211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.193212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.193241 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.193404 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.193405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.193405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.193516 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.193560 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.193752 (+ 192us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.193871 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:14:16.194003 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.182346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:43397 (local address 127.6.68.254:37387)
0503 08:14:16.182411 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.182413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.182597 (+ 184us) server_negotiation.cc:407] Connection header received
0503 08:14:16.183291 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.183292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.183301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.183347 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.184220 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.185001 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.187186 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.187400 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.190353 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.190357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.190357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.190366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.192374 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.192899 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.192900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.192901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.192927 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.193192 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.193193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.193193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.193330 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.193375 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.193762 (+ 387us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.193860 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:14:16.194221 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.182723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:54185 (local address 127.6.68.253:34149)
0503 08:14:16.182803 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.182805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.182814 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:14:16.183415 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.183417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.183428 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.183482 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.185110 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.185774 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.186418 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.186589 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.190352 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.190356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.190357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.190366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.193150 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.193504 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.193506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.193506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.193529 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.193689 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.193690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.193690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.193795 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.193835 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.194018 (+ 183us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.194098 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20240503 08:14:16.194602 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.182594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:34149 (local address 127.6.68.194:54185)
0503 08:14:16.182706 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.182715 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.183047 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.183513 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.183514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.183872 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:16.184209 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.184211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.185938 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.185939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.186326 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.186327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.186478 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.187341 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:16.187365 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:16.190229 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:16.193240 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.193240 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.193241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.193427 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.193597 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.193598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.193599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.193623 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.193882 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:16.193883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:16.193916 (+ 33us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.194310 (+ 394us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.194412 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20240503 08:14:16.194612 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.182325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:43397)
0503 08:14:16.182459 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.182491 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.183033 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.183368 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.183369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.183803 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:16.184118 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.184122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.185096 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.185097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.187097 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.187099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.187157 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.188153 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:16.188157 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:16.190229 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:16.192474 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.192482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.192486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.192805 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.193006 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.193007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.193008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.193118 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.193422 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:16.193427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:16.193598 (+ 171us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.194143 (+ 545us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.194323 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20240503 08:14:16.195112 13687 heartbeater.cc:347] Connected to a master server at 127.6.68.253:34149
I20240503 08:14:16.195114 13686 heartbeater.cc:347] Connected to a master server at 127.6.68.254:37387
I20240503 08:14:16.195227 13686 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:16.195477 13686 heartbeater.cc:511] Master 127.6.68.254:37387 requested a full tablet report, sending...
I20240503 08:14:16.195227 13687 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:16.195752 13687 heartbeater.cc:511] Master 127.6.68.253:34149 requested a full tablet report, sending...
I20240503 08:14:16.195871 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.182324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:38567 (local address 127.6.68.194:45633)
0503 08:14:16.182467 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.182492 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.183047 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.183300 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.183304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.183885 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:16.184209 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.184211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.188231 (+ 4020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.188232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.188885 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.188887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.188965 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.189408 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:16.189412 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:16.190847 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:16.192949 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.192950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.192950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.193122 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.193309 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.193310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.193310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.193336 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.193603 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:16.193604 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:16.193642 (+ 38us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.194623 (+ 981us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.195071 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
I20240503 08:14:16.195902 12572 ts_manager.cc:194] Re-registered known tserver with Master: a91e5b34f54e4586a9eb8c9642ff1108 (127.6.68.194:40553)
I20240503 08:14:16.196043 13173 ts_manager.cc:194] Registered new tserver with Master: a91e5b34f54e4586a9eb8c9642ff1108 (127.6.68.194:40553)
I20240503 08:14:16.196405 12572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:43397
I20240503 08:14:16.196681 13685 heartbeater.cc:347] Connected to a master server at 127.6.68.252:38567
I20240503 08:14:16.196743 13685 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:16.196831 13685 heartbeater.cc:511] Master 127.6.68.252:38567 requested a full tablet report, sending...
I20240503 08:14:16.197124 13377 ts_manager.cc:194] Registered new tserver with Master: a91e5b34f54e4586a9eb8c9642ff1108 (127.6.68.194:40553)
W20240503 08:14:16.277010 13692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:16.277242 13692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:16.277281 13692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:16.278659 13692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:16.278717 13692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:16.278765 13692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:14:16.280504 13692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab
--principal=oryx/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:44211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=36355
--tserver_master_addrs=127.6.68.254:37387,127.6.68.253:34149,127.6.68.252:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:16.280771 13692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:16.281008 13692 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:16.285928 13698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:16.286620 13697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:16.287056 13700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:16.287983 13692 server_base.cc:1034] running on GCE node
I20240503 08:14:16.288197 13692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:16.288425 13692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:16.289577 13692 hybrid_clock.cc:648] HybridClock initialized: now 1714724056289557 us; error 37 us; skew 500 ppm
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:16.292124 13692 init.cc:377] Logged in from keytab as oryx/127.6.68.195@KRBTEST.COM (short username oryx)
I20240503 08:14:16.292630 13692 webserver.cc:460] Webserver started at http://127.6.68.195:36355/ using document root <none> and password file <none>
I20240503 08:14:16.292832 13692 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:16.292907 13692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:16.294201 13692 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:16.294747 13707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.294921 13692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.294997 13692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2"
format_stamp: "Formatted at 2024-05-03 08:14:15 on dist-test-slave-3g67"
server_key: "7c9c1dbb33e3e22eef9dca2acb8f4621"
server_key_iv: "3555b7f7d049fbbd7ccb933b963ae4ff"
server_key_version: "encryptionkey@0"
I20240503 08:14:16.295352 13692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:16.307724 13692 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:16.309983 13692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:16.310125 13692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:16.310371 13692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:16.310669 13692 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:16.310709 13692 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.310745 13692 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:16.310776 13692 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:16.316707 13692 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:44211
I20240503 08:14:16.316738 13820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:44211 every 8 connection(s)
I20240503 08:14:16.317075 13692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.253@KRBTEST.COM
May 03 08:14:16 dist-test-slave-3g67 krb5kdc[12527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724056, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.252@KRBTEST.COM
I20240503 08:14:16.327842 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 13692
I20240503 08:14:16.327989 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.318148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:38321 (local address 127.6.68.253:34149)
0503 08:14:16.318189 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.318191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.318297 (+ 106us) server_negotiation.cc:407] Connection header received
0503 08:14:16.319016 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.319017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.319029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.319069 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.319959 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.320728 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.321511 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.321696 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.324181 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.324186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.324186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.324194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.326279 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.326842 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.326843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.326844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.326869 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.327097 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.327098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.327099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.327230 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.327273 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.327743 (+ 470us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.327858 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:14:16.328912 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.317844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:47383 (local address 127.6.68.252:38567)
0503 08:14:16.317897 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.317899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.318094 (+ 195us) server_negotiation.cc:407] Connection header received
0503 08:14:16.319055 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.319056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.319070 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.319101 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.320030 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.320747 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.321397 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.321580 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.325090 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.325095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.325095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.325103 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.327492 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.327979 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.327980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.327980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.328031 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.328260 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.328284 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.328285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.328416 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.328496 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.328746 (+ 250us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.328831 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240503 08:14:16.328579 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.318045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:34149 (local address 127.6.68.195:38321)
0503 08:14:16.318241 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.318250 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.318934 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.319129 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.319133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.319502 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:16.319870 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.319874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.320826 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.320827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.321385 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.321388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.321488 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.322047 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:16.322051 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:16.324052 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:16.326421 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.326428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.326432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.326742 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.326939 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.326940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.326940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.327023 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.327320 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:16.327325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:16.327548 (+ 223us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.328131 (+ 583us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.328351 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20240503 08:14:16.329816 13825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.317783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:38567 (local address 127.6.68.195:47383)
0503 08:14:16.317970 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.318000 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.318921 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.319176 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.319176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.319583 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:16.319901 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.319904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.320844 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.320845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.321300 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.321302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.321627 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.322280 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:16.322284 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:16.324971 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:16.327593 (+ 2622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.327594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.327595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.327873 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.328141 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.328142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.328142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.328170 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.328529 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:16.328530 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:16.328595 (+ 65us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.329012 (+ 417us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.329139 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20240503 08:14:16.330380 13823 heartbeater.cc:347] Connected to a master server at 127.6.68.253:34149
I20240503 08:14:16.330476 13823 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:16.330646 13821 heartbeater.cc:347] Connected to a master server at 127.6.68.252:38567
I20240503 08:14:16.330693 13821 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:16.330924 13821 heartbeater.cc:511] Master 127.6.68.252:38567 requested a full tablet report, sending...
I20240503 08:14:16.331228 13823 heartbeater.cc:511] Master 127.6.68.253:34149 requested a full tablet report, sending...
I20240503 08:14:16.331344 13377 ts_manager.cc:194] Registered new tserver with Master: 89d4ed61819c4fb88e70a6f87e3ff3a2 (127.6.68.195:44211)
I20240503 08:14:16.331442 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.317859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:35789 (local address 127.6.68.254:37387)
0503 08:14:16.317907 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.317909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.318235 (+ 326us) server_negotiation.cc:407] Connection header received
0503 08:14:16.319055 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.319057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.319070 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.319101 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.320008 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.320507 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.321155 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.321352 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.324849 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.324854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.324854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.324862 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.329177 (+ 4315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.330265 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.330266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.330266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.330310 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.330867 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.330868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.330869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.330991 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.331064 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.331254 (+ 190us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.331357 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240503 08:14:16.331482 13173 ts_manager.cc:194] Registered new tserver with Master: 89d4ed61819c4fb88e70a6f87e3ff3a2 (127.6.68.195:44211)
I20240503 08:14:16.331772 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.317784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:35789)
0503 08:14:16.318161 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.318170 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.318933 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.319176 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.319176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.319583 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:16.319907 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.319908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.320585 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.320585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.321064 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.321065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.321213 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.321868 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:16.321884 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:16.324619 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:16.329258 (+ 4639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.329259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.329260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.330197 (+ 937us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.330763 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:16.330764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:16.330764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:16.330794 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:16.331095 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:16.331096 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:16.331137 (+ 41us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.331519 (+ 382us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.331648 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20240503 08:14:16.332171 13822 heartbeater.cc:347] Connected to a master server at 127.6.68.254:37387
I20240503 08:14:16.332221 13822 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:16.332327 13822 heartbeater.cc:511] Master 127.6.68.254:37387 requested a full tablet report, sending...
I20240503 08:14:16.332659 12572 ts_manager.cc:194] Re-registered known tserver with Master: 89d4ed61819c4fb88e70a6f87e3ff3a2 (127.6.68.195:44211)
I20240503 08:14:16.333216 12572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:35789
I20240503 08:14:16.339919 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.330688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.6.68.252:38567)
0503 08:14:16.330732 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.330734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.332873 (+ 2139us) server_negotiation.cc:407] Connection header received
0503 08:14:16.332960 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.332961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.332971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.333000 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.333920 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.334632 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.335309 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.335472 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.337114 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.337117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.337118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.337125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.338934 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.339214 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.339215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.339215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.339239 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.339378 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.339378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.339379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.339473 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.339578 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.339723 (+ 145us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.339814 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240503 08:14:16.340546 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.330563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.6.68.253:34149)
0503 08:14:16.330605 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.330606 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.331138 (+ 532us) server_negotiation.cc:407] Connection header received
0503 08:14:16.331718 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.331719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.331729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.331757 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.335046 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.335544 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.336259 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.337026 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.337516 (+ 490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.337520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.337520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.337527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.339137 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.339495 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.339496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.339496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.339521 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.339713 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.339714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.339715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.339826 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.339898 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.340364 (+ 466us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.340446 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:14:16.340700 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.329496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.6.68.254:37387)
0503 08:14:16.330046 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.330048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.334053 (+ 4005us) server_negotiation.cc:407] Connection header received
0503 08:14:16.334076 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.334077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.334088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.334152 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.335017 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.335776 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.336457 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.336635 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.337723 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.337727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.337728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.337733 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.339426 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.339773 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.339774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.339775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.339799 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.339964 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.339964 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.339965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.340070 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.340164 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.340501 (+ 337us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.340585 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":107,"threads_started":1}
I20240503 08:14:16.342336 12572 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:14:16.343851 12572 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:14:16.350142 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.347487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:40553 (local address 127.0.0.1:50882)
0503 08:14:16.347805 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.347813 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.347859 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.348108 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.348109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.348112 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:16.348334 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.348337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.349139 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.349140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.349898 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.349900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.349963 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.349968 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.350002 (+ 34us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.350054 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":9,"thread_start_us":87,"threads_started":1}
I20240503 08:14:16.350286 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.347487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:42003 (local address 127.0.0.1:57522)
0503 08:14:16.347616 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.347650 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.347695 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.348217 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.348218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.348220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:16.348412 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.348414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.349287 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.349288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.349997 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.349999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.350068 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.350071 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.350100 (+ 29us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.350136 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20240503 08:14:16.351557 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.347812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.6.68.193:42003)
0503 08:14:16.348039 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.348047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.348056 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:14:16.348097 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.348099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.348116 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.348476 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:16.348536 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.349204 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.350580 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.351175 (+ 595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.351317 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.351426 (+ 109us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.351484 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20240503 08:14:16.352068 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.347538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:44211 (local address 127.0.0.1:48688)
0503 08:14:16.348033 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.348291 (+ 258us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.348349 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.349348 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.349349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.349351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:16.349530 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.349531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.350961 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.350963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.351835 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.351837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.351926 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.351929 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.351964 (+ 35us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.352004 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":104,"threads_started":1}
I20240503 08:14:16.352283 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.347789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.6.68.194:40553)
0503 08:14:16.347946 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.347954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.347962 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:14:16.348004 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.348006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.348025 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.348112 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:16.348411 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.349062 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.351507 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.352054 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.352102 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.352135 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.352188 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20240503 08:14:16.352217 13482 tablet_service.cc:1467] Processing CreateTablet for tablet 84a9cb99050a499bb5142c62b4f905bb (DEFAULT_TABLE table=test-table [id=ed0a0e8ca8c7474baf8f3ffca9cddbfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.352548 13482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a9cb99050a499bb5142c62b4f905bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.352747 13619 tablet_service.cc:1467] Processing CreateTablet for tablet 84a9cb99050a499bb5142c62b4f905bb (DEFAULT_TABLE table=test-table [id=ed0a0e8ca8c7474baf8f3ffca9cddbfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.352838 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.347656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.6.68.195:44211)
0503 08:14:16.347976 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.347984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.348347 (+ 363us) server_negotiation.cc:407] Connection header received
0503 08:14:16.349179 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.349182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.349199 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.349268 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:16.349622 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.350875 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.351968 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.352639 (+ 671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.352681 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.352714 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.352764 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":79,"threads_started":1}
I20240503 08:14:16.352995 13619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a9cb99050a499bb5142c62b4f905bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.355506 13845 tablet_bootstrap.cc:492] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2: Bootstrap starting.
I20240503 08:14:16.355916 13846 tablet_bootstrap.cc:492] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108: Bootstrap starting.
I20240503 08:14:16.356243 13845 tablet_bootstrap.cc:654] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.356288 13755 tablet_service.cc:1467] Processing CreateTablet for tablet 84a9cb99050a499bb5142c62b4f905bb (DEFAULT_TABLE table=test-table [id=ed0a0e8ca8c7474baf8f3ffca9cddbfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:16.356542 13846 tablet_bootstrap.cc:654] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.356590 13755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a9cb99050a499bb5142c62b4f905bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:16.356830 13846 log.cc:824] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:16.356830 13845 log.cc:824] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:16.357591 13846 tablet_bootstrap.cc:492] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108: No bootstrap required, opened a new log
I20240503 08:14:16.357667 13846 ts_tablet_manager.cc:1367] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:16.357860 13845 tablet_bootstrap.cc:492] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2: No bootstrap required, opened a new log
I20240503 08:14:16.357925 13845 ts_tablet_manager.cc:1367] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:16.359535 13849 tablet_bootstrap.cc:492] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2: Bootstrap starting.
I20240503 08:14:16.359546 13846 raft_consensus.cc:348] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40553 } } peers { permanent_uuid: "1d2cb9aab096451d93597c0e6c573ba2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42003 } } peers { permanent_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44211 } }
I20240503 08:14:16.359746 13846 raft_consensus.cc:374] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.359784 13846 raft_consensus.cc:729] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91e5b34f54e4586a9eb8c9642ff1108, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.359694 13845 raft_consensus.cc:348] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40553 } } peers { permanent_uuid: "1d2cb9aab096451d93597c0e6c573ba2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42003 } } peers { permanent_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44211 } }
I20240503 08:14:16.359812 13845 raft_consensus.cc:374] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.359845 13845 raft_consensus.cc:729] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2cb9aab096451d93597c0e6c573ba2, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.359905 13846 consensus_queue.cc:260] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40553 } } peers { permanent_uuid: "1d2cb9aab096451d93597c0e6c573ba2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42003 } } peers { permanent_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44211 } }
I20240503 08:14:16.360152 13846 ts_tablet_manager.cc:1398] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:14:16.360247 13686 heartbeater.cc:503] Master 127.6.68.254:37387 was elected leader, sending a full tablet report...
I20240503 08:14:16.359951 13845 consensus_queue.cc:260] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40553 } } peers { permanent_uuid: "1d2cb9aab096451d93597c0e6c573ba2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42003 } } peers { permanent_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44211 } }
I20240503 08:14:16.360384 13849 tablet_bootstrap.cc:654] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:16.360574 13845 ts_tablet_manager.cc:1398] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:14:16.360672 13849 log.cc:824] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:16.360834 13549 heartbeater.cc:503] Master 127.6.68.254:37387 was elected leader, sending a full tablet report...
I20240503 08:14:16.361402 13849 tablet_bootstrap.cc:492] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2: No bootstrap required, opened a new log
I20240503 08:14:16.361467 13849 ts_tablet_manager.cc:1367] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:16.363171 13849 raft_consensus.cc:348] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40553 } } peers { permanent_uuid: "1d2cb9aab096451d93597c0e6c573ba2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42003 } } peers { permanent_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44211 } }
I20240503 08:14:16.363286 13849 raft_consensus.cc:374] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:16.363312 13849 raft_consensus.cc:729] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d4ed61819c4fb88e70a6f87e3ff3a2, State: Initialized, Role: FOLLOWER
I20240503 08:14:16.363402 13849 consensus_queue.cc:260] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40553 } } peers { permanent_uuid: "1d2cb9aab096451d93597c0e6c573ba2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42003 } } peers { permanent_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44211 } }
I20240503 08:14:16.363601 13849 ts_tablet_manager.cc:1398] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:16.363693 13822 heartbeater.cc:503] Master 127.6.68.254:37387 was elected leader, sending a full tablet report...
I20240503 08:14:16.382371 13851 raft_consensus.cc:482] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:16.382504 13851 raft_consensus.cc:504] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40553 } } peers { permanent_uuid: "1d2cb9aab096451d93597c0e6c573ba2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42003 } } peers { permanent_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44211 } }
I20240503 08:14:16.382881 13851 leader_election.cc:288] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91e5b34f54e4586a9eb8c9642ff1108 (127.6.68.194:40553), 89d4ed61819c4fb88e70a6f87e3ff3a2 (127.6.68.195:44211)
I20240503 08:14:16.385525 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.383094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:44211 (local address 127.6.68.193:37911)
0503 08:14:16.383144 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.383156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.383207 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.383361 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.383362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.383364 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:16.383646 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.383649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.384663 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.384664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.385294 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.385296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.385367 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.385371 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.385404 (+ 33us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.385441 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20240503 08:14:16.385527 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.383101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:40553 (local address 127.6.68.193:51277)
0503 08:14:16.383209 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:16.383218 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:16.383248 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:16.383394 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:16.383395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:16.383396 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:16.383656 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.383657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.384622 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.384624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:16.385305 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:16.385307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.385381 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.385385 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:14:16.385416 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:14:16.385453 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":68}
I20240503 08:14:16.385988 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.383166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:37911 (local address 127.6.68.195:44211)
0503 08:14:16.383225 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.383227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.383238 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:14:16.383274 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.383274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.383285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.383354 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:16.383751 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.384542 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.385388 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.385892 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.385899 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.385918 (+ 19us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.385942 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:14:16.385998 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.383248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:51277 (local address 127.6.68.194:40553)
0503 08:14:16.383290 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.383292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.383299 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:14:16.383323 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.383324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.383333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.383388 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:16.383734 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.384542 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.385400 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.385872 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.385881 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.385903 (+ 22us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.385937 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240503 08:14:16.386318 13775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a9cb99050a499bb5142c62b4f905bb" candidate_uuid: "1d2cb9aab096451d93597c0e6c573ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" is_pre_election: true
I20240503 08:14:16.386345 13639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a9cb99050a499bb5142c62b4f905bb" candidate_uuid: "1d2cb9aab096451d93597c0e6c573ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" is_pre_election: true
I20240503 08:14:16.386459 13775 raft_consensus.cc:2454] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d2cb9aab096451d93597c0e6c573ba2 in term 0.
I20240503 08:14:16.386472 13639 raft_consensus.cc:2454] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d2cb9aab096451d93597c0e6c573ba2 in term 0.
I20240503 08:14:16.386664 13437 leader_election.cc:302] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d2cb9aab096451d93597c0e6c573ba2, a91e5b34f54e4586a9eb8c9642ff1108; no voters:
I20240503 08:14:16.386816 13851 raft_consensus.cc:2781] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:16.386873 13851 raft_consensus.cc:482] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:16.386888 13851 raft_consensus.cc:3037] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:16.387826 13851 raft_consensus.cc:504] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40553 } } peers { permanent_uuid: "1d2cb9aab096451d93597c0e6c573ba2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42003 } } peers { permanent_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44211 } }
I20240503 08:14:16.387976 13851 leader_election.cc:288] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [CANDIDATE]: Term 1 election: Requested vote from peers a91e5b34f54e4586a9eb8c9642ff1108 (127.6.68.194:40553), 89d4ed61819c4fb88e70a6f87e3ff3a2 (127.6.68.195:44211)
I20240503 08:14:16.388141 13775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a9cb99050a499bb5142c62b4f905bb" candidate_uuid: "1d2cb9aab096451d93597c0e6c573ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2"
I20240503 08:14:16.388154 13639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a9cb99050a499bb5142c62b4f905bb" candidate_uuid: "1d2cb9aab096451d93597c0e6c573ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91e5b34f54e4586a9eb8c9642ff1108"
I20240503 08:14:16.388214 13639 raft_consensus.cc:3037] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:16.388211 13775 raft_consensus.cc:3037] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:16.388870 13775 raft_consensus.cc:2454] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2cb9aab096451d93597c0e6c573ba2 in term 1.
I20240503 08:14:16.388921 13639 raft_consensus.cc:2454] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2cb9aab096451d93597c0e6c573ba2 in term 1.
I20240503 08:14:16.389067 13435 leader_election.cc:302] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d2cb9aab096451d93597c0e6c573ba2, 89d4ed61819c4fb88e70a6f87e3ff3a2; no voters:
I20240503 08:14:16.389158 13851 raft_consensus.cc:2781] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:16.389312 13851 raft_consensus.cc:686] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 1 LEADER]: Becoming Leader. State: Replica: 1d2cb9aab096451d93597c0e6c573ba2, State: Running, Role: LEADER
I20240503 08:14:16.389396 13851 consensus_queue.cc:237] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40553 } } peers { permanent_uuid: "1d2cb9aab096451d93597c0e6c573ba2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42003 } } peers { permanent_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44211 } }
I20240503 08:14:16.390496 12572 catalog_manager.cc:5484] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d2cb9aab096451d93597c0e6c573ba2 (127.6.68.193). New cstate: current_term: 1 leader_uuid: "1d2cb9aab096451d93597c0e6c573ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d2cb9aab096451d93597c0e6c573ba2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44211 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:16.398383 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.396298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.6.68.193:42003)
0503 08:14:16.396364 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.396366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.396377 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:14:16.396426 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.396427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.396444 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.396513 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:14:16.396781 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.397294 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.397868 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.398015 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.398053 (+ 38us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:14:16.398136 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:14:16.398209 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.398270 (+ 61us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.398296 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:14:16.400102 13639 raft_consensus.cc:1261] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108 [term 1 FOLLOWER]: Refusing update from remote peer 1d2cb9aab096451d93597c0e6c573ba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:16.400131 13775 raft_consensus.cc:1261] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2 [term 1 FOLLOWER]: Refusing update from remote peer 1d2cb9aab096451d93597c0e6c573ba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:16.400449 13854 consensus_queue.cc:1035] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91e5b34f54e4586a9eb8c9642ff1108" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:16.400588 13851 consensus_queue.cc:1035] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d4ed61819c4fb88e70a6f87e3ff3a2" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:16.403606 13857 mvcc.cc:205] Tried to move back new op lower bound from 7023509735012171776 to 7023509734971654144. Current Snapshot: MvccSnapshot[applied={T|T < 7023509735012171776}]
I20240503 08:14:16.405897 12572 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57178:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:14:16.405961 13859 mvcc.cc:205] Tried to move back new op lower bound from 7023509735012171776 to 7023509734971654144. Current Snapshot: MvccSnapshot[applied={T|T < 7023509735012171776}]
I20240503 08:14:16.406021 12572 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57178:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:14:16.407609 12572 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 39d8111f3ddf436a859b99d7496b0ea0: Sending DeleteTablet for 3 replicas of tablet 84a9cb99050a499bb5142c62b4f905bb
I20240503 08:14:16.408022 13482 tablet_service.cc:1514] Processing DeleteTablet for tablet 84a9cb99050a499bb5142c62b4f905bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:16 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:57522
I20240503 08:14:16.408005 13619 tablet_service.cc:1514] Processing DeleteTablet for tablet 84a9cb99050a499bb5142c62b4f905bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:16 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:50882
I20240503 08:14:16.408272 13868 tablet_replica.cc:330] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108: stopping tablet replica
I20240503 08:14:16.408243 13755 tablet_service.cc:1514] Processing DeleteTablet for tablet 84a9cb99050a499bb5142c62b4f905bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:16 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:48688
I20240503 08:14:16.408360 13868 raft_consensus.cc:2229] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:16.408454 13868 raft_consensus.cc:2258] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:16.408602 13867 tablet_replica.cc:330] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2: stopping tablet replica
I20240503 08:14:16.408668 13867 raft_consensus.cc:2229] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:14:16.408746 13867 raft_consensus.cc:2258] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:16.409202 13867 ts_tablet_manager.cc:1875] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:16.409868 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 12540
I20240503 08:14:16.410674 13868 ts_tablet_manager.cc:1875] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:16.412309 13868 ts_tablet_manager.cc:1888] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:14:16.412389 13868 log.cc:1196] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/84a9cb99050a499bb5142c62b4f905bb
I20240503 08:14:16.412668 13868 ts_tablet_manager.cc:1909] T 84a9cb99050a499bb5142c62b4f905bb P a91e5b34f54e4586a9eb8c9642ff1108: Deleting consensus metadata
I20240503 08:14:16.413750 13867 ts_tablet_manager.cc:1888] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:14:16.413837 13867 log.cc:1196] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/84a9cb99050a499bb5142c62b4f905bb
I20240503 08:14:16.414116 13867 ts_tablet_manager.cc:1909] T 84a9cb99050a499bb5142c62b4f905bb P 1d2cb9aab096451d93597c0e6c573ba2: Deleting consensus metadata
I20240503 08:14:16.416818 13869 tablet_replica.cc:330] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2: stopping tablet replica
I20240503 08:14:16.416940 13869 raft_consensus.cc:2229] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:16.417052 13869 raft_consensus.cc:2258] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:16.417587 13869 ts_tablet_manager.cc:1875] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:14:16.418022 13436 connection.cc:535] server connection from 127.0.0.1:57522 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:14:16.418272 13572 connection.cc:535] server connection from 127.0.0.1:50882 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:14:16.419246 13869 ts_tablet_manager.cc:1888] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:14:16.419318 13869 log.cc:1196] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/84a9cb99050a499bb5142c62b4f905bb
I20240503 08:14:16.419582 13869 ts_tablet_manager.cc:1909] T 84a9cb99050a499bb5142c62b4f905bb P 89d4ed61819c4fb88e70a6f87e3ff3a2: Deleting consensus metadata
W20240503 08:14:16.420082 13708 connection.cc:441] server connection from 127.0.0.1:48688 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48688 (request call id 1) could send its response
I20240503 08:14:16.420337 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.408916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.6.68.252:38567)
0503 08:14:16.408951 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.408954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.408962 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:14:16.409012 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.409013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.409024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.409053 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.410272 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.411047 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.412021 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.412236 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.416734 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.416740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.416740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.416750 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.418693 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.419102 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.419103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.419103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.419128 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.419290 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.419291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.419291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.419412 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.419510 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.420099 (+ 589us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.420193 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240503 08:14:16.427868 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.420642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.6.68.252:38567)
0503 08:14:16.420678 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.420680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.421001 (+ 321us) server_negotiation.cc:407] Connection header received
0503 08:14:16.421098 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.421099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.421109 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.421139 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.421814 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.422298 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.423069 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.423255 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.424609 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.424612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.424612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.424618 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.426379 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.426745 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.426746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.426746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.426774 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.426979 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.426980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.426980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.427104 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.427208 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.427561 (+ 353us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.427762 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:14:16.428444 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:16.420467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.6.68.253:34149)
0503 08:14:16.420509 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:16.420511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:16.420821 (+ 310us) server_negotiation.cc:407] Connection header received
0503 08:14:16.420954 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:16.420955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:16.420966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:16.420995 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:16.422443 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.423128 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:16.423837 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:16.424341 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:16.424957 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:16.424963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:16.424964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:16.424970 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:16.426652 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.427115 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.427116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.427116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.427140 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:16.427296 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:16.427297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:16.427297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:16.427416 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:16.427508 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:16.428258 (+ 750us) server_negotiation.cc:299] Negotiation successful
0503 08:14:16.428347 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:14:17.405937 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:17.405668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:34617)
0503 08:14:17.405809 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:17.405873 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20240503 08:14:17.406186 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:17.405674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:46077)
0503 08:14:17.406089 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:17.406137 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":73,"threads_started":1}
W20240503 08:14:17.406351 13549 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:37387 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
W20240503 08:14:17.406353 13686 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:37387 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
I20240503 08:14:17.408723 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:17.408528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:32831)
0503 08:14:17.408634 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:17.408685 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
W20240503 08:14:17.408896 13822 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:37387 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
I20240503 08:14:17.907850 13881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39d8111f3ddf436a859b99d7496b0ea0)
I20240503 08:14:17.907980 13881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
I20240503 08:14:17.908205 13881 leader_election.cc:288] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387)
I20240503 08:14:17.908586 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:17.908368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:57642)
0503 08:14:17.908489 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:17.908546 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
W20240503 08:14:17.908752 13158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
I20240503 08:14:17.909174 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:17.909088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:57654)
0503 08:14:17.909133 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:17.909156 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240503 08:14:17.909258 13158 leader_election.cc:334] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387): Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
I20240503 08:14:17.909322 13158 leader_election.cc:302] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 7b64c951cd374095b0772627949c3b75; no voters: 39d8111f3ddf436a859b99d7496b0ea0
I20240503 08:14:17.909456 13881 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:18.407060 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:18.406747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:37469)
0503 08:14:18.406944 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:18.406988 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
I20240503 08:14:18.407063 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:18.406748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:49325)
0503 08:14:18.406973 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:18.407011 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
I20240503 08:14:18.409427 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:18.409217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:43693)
0503 08:14:18.409347 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:18.409385 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20240503 08:14:19.407846 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:19.407548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:46183)
0503 08:14:19.407780 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:19.407808 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
I20240503 08:14:19.407842 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:19.407587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:46125)
0503 08:14:19.407732 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:19.407779 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20240503 08:14:19.410068 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:19.409852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:54645)
0503 08:14:19.409978 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:19.410016 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20240503 08:14:20.031559 13890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39d8111f3ddf436a859b99d7496b0ea0)
I20240503 08:14:20.031683 13890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
I20240503 08:14:20.031843 13890 leader_election.cc:288] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387)
I20240503 08:14:20.032176 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:20.031999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:47112)
0503 08:14:20.032103 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:20.032137 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20240503 08:14:20.032686 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:20.032588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:47122)
0503 08:14:20.032633 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:20.032659 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240503 08:14:20.032778 13158 leader_election.cc:334] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387): Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
I20240503 08:14:20.032816 13158 leader_election.cc:302] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 7b64c951cd374095b0772627949c3b75; no voters: 39d8111f3ddf436a859b99d7496b0ea0
I20240503 08:14:20.032876 13890 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:20.408659 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:20.408312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:48643)
0503 08:14:20.408548 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:20.408591 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":66,"threads_started":1}
I20240503 08:14:20.408958 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:20.408350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:56091)
0503 08:14:20.408869 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:20.408908 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":67,"threads_started":1}
I20240503 08:14:20.410751 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:20.410536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:44085)
0503 08:14:20.410668 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:20.410703 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20240503 08:14:21.409632 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:21.409254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:56919)
0503 08:14:21.409523 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:21.409564 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":77,"threads_started":1}
I20240503 08:14:21.409736 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:21.409431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:36105)
0503 08:14:21.409644 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:21.409683 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":64,"threads_started":1}
I20240503 08:14:21.411314 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:21.411140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:44849)
0503 08:14:21.411239 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:21.411274 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":51,"threads_started":1}
I20240503 08:14:22.010076 13899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39d8111f3ddf436a859b99d7496b0ea0)
I20240503 08:14:22.010186 13899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
I20240503 08:14:22.010406 13899 leader_election.cc:288] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387)
I20240503 08:14:22.010767 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:22.010581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:47308)
0503 08:14:22.010687 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:22.010724 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
I20240503 08:14:22.011190 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:22.011092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:47322)
0503 08:14:22.011148 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:22.011173 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20240503 08:14:22.011265 13158 leader_election.cc:334] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387): Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
I20240503 08:14:22.011309 13158 leader_election.cc:302] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 7b64c951cd374095b0772627949c3b75; no voters: 39d8111f3ddf436a859b99d7496b0ea0
I20240503 08:14:22.011391 13899 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:22.410645 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:22.410167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:51315)
0503 08:14:22.410537 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:22.410583 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":83,"threads_started":1}
I20240503 08:14:22.410753 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:22.410129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:43865)
0503 08:14:22.410674 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:22.410708 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":85,"threads_started":1}
I20240503 08:14:22.411940 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:22.411750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:53505)
0503 08:14:22.411870 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:22.411900 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20240503 08:14:23.411446 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:23.411153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:46815)
0503 08:14:23.411338 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:23.411379 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":56,"threads_started":1}
I20240503 08:14:23.411487 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:23.411136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:33633)
0503 08:14:23.411393 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:23.411430 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
I20240503 08:14:23.412441 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:23.412290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:48145)
0503 08:14:23.412380 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:23.412403 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":42,"threads_started":1}
I20240503 08:14:24.412338 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:24.412064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:55141)
0503 08:14:24.412227 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:24.412268 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20240503 08:14:24.412497 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:24.412251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:57735)
0503 08:14:24.412420 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:24.412451 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20240503 08:14:24.412994 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:24.412826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:50765)
0503 08:14:24.412924 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:24.412958 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":44,"threads_started":1}
I20240503 08:14:25.413479 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:25.413064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:33045)
0503 08:14:25.413241 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:25.413416 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":65,"threads_started":1}
I20240503 08:14:25.413556 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:25.413064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:38179)
0503 08:14:25.413481 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:25.413512 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":76,"threads_started":1}
I20240503 08:14:25.413635 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:25.413370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:44709)
0503 08:14:25.413488 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:25.413528 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":52,"threads_started":1}
I20240503 08:14:26.373409 13914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39d8111f3ddf436a859b99d7496b0ea0)
I20240503 08:14:26.373528 13914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
I20240503 08:14:26.373713 13914 leader_election.cc:288] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387)
I20240503 08:14:26.374042 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:26.373851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:47666)
0503 08:14:26.373966 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:26.374000 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":41,"threads_started":1}
W20240503 08:14:26.374126 13158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20240503 08:14:26.374478 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:26.374383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:47678)
0503 08:14:26.374433 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:26.374456 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20240503 08:14:26.374547 13158 leader_election.cc:334] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387): Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
I20240503 08:14:26.374586 13158 leader_election.cc:302] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 7b64c951cd374095b0772627949c3b75; no voters: 39d8111f3ddf436a859b99d7496b0ea0
I20240503 08:14:26.374650 13914 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:26.414153 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:26.413934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:41845)
0503 08:14:26.414060 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:26.414096 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20240503 08:14:26.414186 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:26.413918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:41439)
0503 08:14:26.414064 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:26.414096 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20240503 08:14:26.414222 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:26.414008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:40433)
0503 08:14:26.414137 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:26.414174 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20240503 08:14:27.414984 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:27.414594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:52279)
0503 08:14:27.414871 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:27.414916 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20240503 08:14:27.415045 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:27.414742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:59641)
0503 08:14:27.414955 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:27.414993 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":62,"threads_started":1}
I20240503 08:14:27.415171 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:27.414659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:59675)
0503 08:14:27.415054 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:27.415088 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":59,"threads_started":1}
I20240503 08:14:28.415738 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:28.415449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:50351)
0503 08:14:28.415600 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:28.415673 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20240503 08:14:28.416010 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:28.415705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:48267)
0503 08:14:28.415930 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:28.415962 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":50,"threads_started":1}
I20240503 08:14:28.416010 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:28.415705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:52327)
0503 08:14:28.415930 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:28.415962 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":48,"threads_started":1}
I20240503 08:14:29.416455 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:29.416151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:49611)
0503 08:14:29.416330 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:29.416388 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20240503 08:14:29.416628 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:29.416430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:45957)
0503 08:14:29.416526 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:29.416557 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":51,"threads_started":1}
I20240503 08:14:29.416695 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:29.416413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:50649)
0503 08:14:29.416612 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:29.416651 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":55,"threads_started":1}
I20240503 08:14:30.417263 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:30.417032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:37913)
0503 08:14:30.417159 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:30.417201 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20240503 08:14:30.417376 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:30.417032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:57219)
0503 08:14:30.417277 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:30.417316 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":64,"threads_started":1}
I20240503 08:14:30.417464 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:30.417219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:36837)
0503 08:14:30.417383 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:30.417418 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20240503 08:14:31.418129 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:31.417913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:45817)
0503 08:14:31.418030 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:31.418074 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20240503 08:14:31.418128 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:31.417877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:53177)
0503 08:14:31.418041 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:31.418073 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20240503 08:14:31.418128 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:31.417877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:45811)
0503 08:14:31.418030 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:31.418073 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20240503 08:14:32.418860 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:32.418593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:42753)
0503 08:14:32.418740 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:32.418814 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20240503 08:14:32.418841 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:32.418593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:55713)
0503 08:14:32.418741 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:32.418783 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20240503 08:14:32.418977 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:32.418608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:48907)
0503 08:14:32.418885 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:32.418921 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":59,"threads_started":1}
I20240503 08:14:32.540231 13938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39d8111f3ddf436a859b99d7496b0ea0)
I20240503 08:14:32.540424 13938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
I20240503 08:14:32.540650 13938 leader_election.cc:288] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387)
I20240503 08:14:32.543051 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:32.542770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:34628)
0503 08:14:32.542939 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:32.542984 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
W20240503 08:14:32.543215 13158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
I20240503 08:14:32.543743 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:32.543592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:34636)
0503 08:14:32.543663 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:32.543709 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20240503 08:14:32.543852 13158 leader_election.cc:334] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387): Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
I20240503 08:14:32.543900 13158 leader_election.cc:302] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 7b64c951cd374095b0772627949c3b75; no voters: 39d8111f3ddf436a859b99d7496b0ea0
I20240503 08:14:32.543972 13938 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:33.419685 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:33.419389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:42417)
0503 08:14:33.419549 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:33.419593 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
I20240503 08:14:33.419677 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:33.419393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:53981)
0503 08:14:33.419552 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:33.419594 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":61,"threads_started":1}
I20240503 08:14:33.419687 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:33.419466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:59289)
0503 08:14:33.419595 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:33.419646 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20240503 08:14:34.420454 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:34.420170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:60717)
0503 08:14:34.420363 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:34.420398 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
I20240503 08:14:34.420454 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:34.420199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:54035)
0503 08:14:34.420354 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:34.420397 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20240503 08:14:34.420500 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:34.420170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:54397)
0503 08:14:34.420406 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:34.420443 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":65,"threads_started":1}
I20240503 08:14:35.421183 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:35.420872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:48581)
0503 08:14:35.421077 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:35.421117 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":64,"threads_started":1}
I20240503 08:14:35.421326 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:35.421107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:47269)
0503 08:14:35.421240 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:35.421279 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20240503 08:14:35.421326 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:35.421103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:45793)
0503 08:14:35.421249 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:35.421280 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20240503 08:14:36.421906 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:36.421671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:37439)
0503 08:14:36.421799 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:36.421841 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20240503 08:14:36.421938 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:36.421742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:40819)
0503 08:14:36.421859 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:36.421890 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20240503 08:14:36.421938 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:36.421742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:42833)
0503 08:14:36.421859 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:36.421890 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20240503 08:14:36.507771 13953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39d8111f3ddf436a859b99d7496b0ea0)
I20240503 08:14:36.507862 13953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39d8111f3ddf436a859b99d7496b0ea0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37387 } } peers { permanent_uuid: "7b64c951cd374095b0772627949c3b75" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "7d6c48594ed342ed81acbbe20139375d" member_type: NON_VOTER last_known_addr { host: "127.6.68.252" port: 38567 } attrs { promote: true } }
I20240503 08:14:36.508031 13953 leader_election.cc:288] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387)
I20240503 08:14:36.508375 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:36.508193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:34824)
0503 08:14:36.508292 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:36.508329 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
I20240503 08:14:36.508745 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:36.508657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.0.0.1:34828)
0503 08:14:36.508701 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:36.508723 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240503 08:14:36.508814 13158 leader_election.cc:334] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 39d8111f3ddf436a859b99d7496b0ea0 (127.6.68.254:37387): Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
I20240503 08:14:36.508864 13158 leader_election.cc:302] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 7b64c951cd374095b0772627949c3b75; no voters: 39d8111f3ddf436a859b99d7496b0ea0
I20240503 08:14:36.508950 13953 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7b64c951cd374095b0772627949c3b75 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:14:37.422607 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:37.422383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:37701)
0503 08:14:37.422501 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:37.422541 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20240503 08:14:37.422648 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:37.422408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:42157)
0503 08:14:37.422561 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:37.422595 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":54,"threads_started":1}
I20240503 08:14:37.422786 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:37.422545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:54735)
0503 08:14:37.422693 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:37.422730 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":43,"threads_started":1}
I20240503 08:14:38.423362 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:38.423182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:34641)
0503 08:14:38.423296 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:38.423324 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20240503 08:14:38.423355 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:38.423129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:58529)
0503 08:14:38.423255 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:38.423296 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20240503 08:14:38.423388 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:38.423233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:53319)
0503 08:14:38.423316 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:38.423346 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":43,"threads_started":1}
I20240503 08:14:39.424055 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:39.423775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:53677)
0503 08:14:39.423931 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:39.423989 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20240503 08:14:39.424072 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:39.423848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:35189)
0503 08:14:39.423981 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:39.424016 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20240503 08:14:39.424103 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:39.423932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:37693)
0503 08:14:39.424037 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:39.424060 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20240503 08:14:40.424865 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:40.424612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:41859)
0503 08:14:40.424763 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:40.424804 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20240503 08:14:40.424906 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:40.424702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:44081)
0503 08:14:40.424821 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:40.424854 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":51,"threads_started":1}
I20240503 08:14:40.425043 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:40.424877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:46881)
0503 08:14:40.424969 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:40.424997 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":43,"threads_started":1}
I20240503 08:14:41.425696 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:41.425448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:35395)
0503 08:14:41.425577 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:41.425625 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20240503 08:14:41.425776 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:41.425466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:52907)
0503 08:14:41.425703 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:41.425732 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":55,"threads_started":1}
I20240503 08:14:41.425866 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:41.425622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:51483)
0503 08:14:41.425777 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:41.425812 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":42,"threads_started":1}
I20240503 08:14:42.426529 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:42.426275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:54091)
0503 08:14:42.426439 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:42.426471 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
I20240503 08:14:42.426529 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:42.426275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:37113)
0503 08:14:42.426430 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:42.426471 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20240503 08:14:42.426738 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:42.426331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:60671)
0503 08:14:42.426611 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:42.426651 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":44,"threads_started":1}
I20240503 08:14:43.427165 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:43.426902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:38123)
0503 08:14:43.427080 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:43.427115 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":87,"threads_started":1}
I20240503 08:14:43.427351 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:43.427087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:49331)
0503 08:14:43.427261 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:43.427297 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":53,"threads_started":1}
I20240503 08:14:43.427397 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:43.427087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:53433)
0503 08:14:43.427323 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:43.427350 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":55,"threads_started":1}
I20240503 08:14:44.428015 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:44.427791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:59419)
0503 08:14:44.427923 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:44.427962 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":45,"threads_started":1}
I20240503 08:14:44.427975 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:44.427705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:39589)
0503 08:14:44.427874 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:44.427914 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20240503 08:14:44.428191 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:44.427806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:56911)
0503 08:14:44.428109 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:44.428144 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":56,"threads_started":1}
I20240503 08:14:45.428635 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:45.428402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:46223)
0503 08:14:45.428562 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:45.428591 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20240503 08:14:45.428635 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:45.428435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:43213)
0503 08:14:45.428538 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:45.428580 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":56,"threads_started":1}
I20240503 08:14:45.428812 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:45.428622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:40805)
0503 08:14:45.428722 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:45.428755 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":49,"threads_started":1}
I20240503 08:14:46.429365 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:46.429126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.195:49757)
0503 08:14:46.429265 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:46.429315 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20240503 08:14:46.429407 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:46.429201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.194:43259)
0503 08:14:46.429324 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:46.429359 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20240503 08:14:46.429445 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:46.429232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37387 (local address 127.6.68.193:53089)
0503 08:14:46.429332 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:46.429359 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":45,"threads_started":1}
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.6.68.254:37387,127.6.68.253:34149,127.6.68.252:38567, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.6.68.254:37387,127.6.68.253:34149,127.6.68.252:38567, num_attempts: 1)
I20240503 08:14:46.602350 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 13320
I20240503 08:14:46.608475 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 13556
I20240503 08:14:46.613438 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 13692
I20240503 08:14:46.618183 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 13036
I20240503 08:14:46.622990 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 13227
2024-05-03T08:14:46Z chronyd exiting
I20240503 08:14:46.639353 6419 test_util.cc:175] -----------------------------------------------
I20240503 08:14:46.639420 6419 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714724013808011-6419-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33073 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:14:46 dist-test-slave-3g67 krb5kdc[13988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:14:46 dist-test-slave-3g67 krb5kdc[13988](info): set up 2 sockets
May 03 08:14:46 dist-test-slave-3g67 krb5kdc[13988](info): commencing operation
krb5kdc: starting...
W20240503 08:14:47.564309 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:14:47 dist-test-slave-3g67 krb5kdc[13988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:14:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:14:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.254.keytab.
Entry for principal oryx/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.254.keytab.
I20240503 08:14:47.679687 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:43581
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.254.keytab
--principal=oryx/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:44847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:43581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:47.757266 14001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:47.757447 14001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:47.757467 14001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:47.758745 14001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:47.758791 14001 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240503 08:14:47.758805 14001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:47.758816 14001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:47.758828 14001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:47.758839 14001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:14:47.760520 14001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:44847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:43581
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.254.keytab
--principal=oryx/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:43581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:47.760716 14001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:47.760926 14001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:47.765576 14006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:47.766168 14009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:47.766772 14007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:47.767743 14001 server_base.cc:1034] running on GCE node
I20240503 08:14:47.767982 14001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:47.768210 14001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:47.769344 14001 hybrid_clock.cc:648] HybridClock initialized: now 1714724087769329 us; error 33 us; skew 500 ppm
May 03 08:14:47 dist-test-slave-3g67 krb5kdc[13988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724087, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:47.771572 14001 init.cc:377] Logged in from keytab as oryx/127.6.68.254@KRBTEST.COM (short username oryx)
I20240503 08:14:47.771984 14001 webserver.cc:460] Webserver started at http://127.6.68.254:45735/ using document root <none> and password file <none>
I20240503 08:14:47.772150 14001 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:47.772192 14001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:47.772307 14001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:47.773290 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "797d178d01474365bcbd22e5304f8744"
format_stamp: "Formatted at 2024-05-03 08:14:47 on dist-test-slave-3g67"
server_key: "a3fc12b46240441a1d2edf24536da284"
server_key_iv: "6f63541b75da34c6019563cd42cfbe1f"
server_key_version: "encryptionkey@0"
I20240503 08:14:47.773588 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "797d178d01474365bcbd22e5304f8744"
format_stamp: "Formatted at 2024-05-03 08:14:47 on dist-test-slave-3g67"
server_key: "a3fc12b46240441a1d2edf24536da284"
server_key_iv: "6f63541b75da34c6019563cd42cfbe1f"
server_key_version: "encryptionkey@0"
I20240503 08:14:47.774698 14001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:14:47.775290 14016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:47.775457 14001 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:47.775529 14001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "797d178d01474365bcbd22e5304f8744"
format_stamp: "Formatted at 2024-05-03 08:14:47 on dist-test-slave-3g67"
server_key: "a3fc12b46240441a1d2edf24536da284"
server_key_iv: "6f63541b75da34c6019563cd42cfbe1f"
server_key_version: "encryptionkey@0"
I20240503 08:14:47.775583 14001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:47.817946 14001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:47.820041 14001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:47.820166 14001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:47.824215 14001 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:43581
I20240503 08:14:47.824455 14078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:43581 every 8 connection(s)
I20240503 08:14:47.824553 14001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:14:47.825109 14079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:47.827093 14079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744: Bootstrap starting.
I20240503 08:14:47.827610 14079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:47.827883 14079 log.cc:824] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:47.828531 14079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744: No bootstrap required, opened a new log
I20240503 08:14:47.829859 14079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797d178d01474365bcbd22e5304f8744" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43581 } }
I20240503 08:14:47.829974 14079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:47.829991 14079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797d178d01474365bcbd22e5304f8744, State: Initialized, Role: FOLLOWER
I20240503 08:14:47.830098 14079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797d178d01474365bcbd22e5304f8744" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43581 } }
I20240503 08:14:47.830159 14079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:14:47.830180 14079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:14:47.830205 14079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:47.830977 14079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797d178d01474365bcbd22e5304f8744" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43581 } }
I20240503 08:14:47.831061 14079 leader_election.cc:302] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 797d178d01474365bcbd22e5304f8744; no voters:
I20240503 08:14:47.831199 14079 leader_election.cc:288] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:14:47.831262 14082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:47.831418 14079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:47.831435 14082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [term 1 LEADER]: Becoming Leader. State: Replica: 797d178d01474365bcbd22e5304f8744, State: Running, Role: LEADER
I20240503 08:14:47.831532 14082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797d178d01474365bcbd22e5304f8744" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43581 } }
I20240503 08:14:47.831912 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 797d178d01474365bcbd22e5304f8744. Latest consensus state: current_term: 1 leader_uuid: "797d178d01474365bcbd22e5304f8744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797d178d01474365bcbd22e5304f8744" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43581 } } }
I20240503 08:14:47.831924 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "797d178d01474365bcbd22e5304f8744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797d178d01474365bcbd22e5304f8744" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43581 } } }
I20240503 08:14:47.832113 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:47.832132 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744 [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:47.832296 14091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:14:47.832760 14091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:14:47.834079 14091 catalog_manager.cc:1347] Generated new cluster ID: ffb9197fb03f442888ac95391f3abe4a
I20240503 08:14:47.834126 14091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:14:47.835846 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 14001
I20240503 08:14:47.835942 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:14:47.836107 6419 external_mini_cluster.cc:1428] Setting key 89d6389e486a6e303704f50e794788ae
I20240503 08:14:47.840342 14091 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:14:47.840960 14091 catalog_manager.cc:1504] Loading token signing keys...
May 03 08:14:47 dist-test-slave-3g67 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:47.846055 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:47.836556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.6.68.254:43581)
0503 08:14:47.836717 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:47.836728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:47.836802 (+ 74us) server_negotiation.cc:407] Connection header received
0503 08:14:47.836933 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:47.836938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:47.836984 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:47.837142 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:47.837969 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:47.839184 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:47.839953 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:47.840241 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:47.842802 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:47.842813 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:47.842813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:47.842831 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:47.844762 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:47.845118 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:47.845120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:47.845120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:47.845167 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:47.845299 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:47.845299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:47.845300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:47.845494 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:47.845605 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:47.845734 (+ 129us) server_negotiation.cc:299] Negotiation successful
0503 08:14:47.845865 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20240503 08:14:47.848620 14091 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744: Generated new TSK 0
I20240503 08:14:47.848809 14091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab.
Entry for principal oryx/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab.
I20240503 08:14:47.882133 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43581
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab
--principal=oryx/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:44847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:47.994459 14105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:47.994688 14105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:47.994724 14105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:47.996624 14105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:47.996693 14105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:47.996723 14105 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240503 08:14:47.996762 14105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:14:47.999230 14105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:44847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab
--principal=oryx/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.68.254:43581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:47.999468 14105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:47.999692 14105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:48.004568 14111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:48.004949 14113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:48.005640 14110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:48.006486 14105 server_base.cc:1034] running on GCE node
I20240503 08:14:48.006681 14105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:48.006917 14105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:48.008074 14105 hybrid_clock.cc:648] HybridClock initialized: now 1714724088008059 us; error 32 us; skew 500 ppm
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724088, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:48.011018 14105 init.cc:377] Logged in from keytab as oryx/127.6.68.193@KRBTEST.COM (short username oryx)
I20240503 08:14:48.011533 14105 webserver.cc:460] Webserver started at http://127.6.68.193:39423/ using document root <none> and password file <none>
I20240503 08:14:48.011785 14105 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:48.011845 14105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:48.011972 14105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:48.013243 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "d4be1c81b23d41d59b3af62afdf5c4a4"
format_stamp: "Formatted at 2024-05-03 08:14:48 on dist-test-slave-3g67"
server_key: "a6c18cbd3ea29c838b887649bce2f28d"
server_key_iv: "698af94b43356938a67f1ebfc737dd65"
server_key_version: "encryptionkey@0"
I20240503 08:14:48.013614 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "d4be1c81b23d41d59b3af62afdf5c4a4"
format_stamp: "Formatted at 2024-05-03 08:14:48 on dist-test-slave-3g67"
server_key: "a6c18cbd3ea29c838b887649bce2f28d"
server_key_iv: "698af94b43356938a67f1ebfc737dd65"
server_key_version: "encryptionkey@0"
I20240503 08:14:48.015163 14105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:14:48.015964 14120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.016166 14105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.016335 14105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "d4be1c81b23d41d59b3af62afdf5c4a4"
format_stamp: "Formatted at 2024-05-03 08:14:48 on dist-test-slave-3g67"
server_key: "a6c18cbd3ea29c838b887649bce2f28d"
server_key_iv: "698af94b43356938a67f1ebfc737dd65"
server_key_version: "encryptionkey@0"
I20240503 08:14:48.016423 14105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:48.030536 14105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:48.032832 14105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:48.032976 14105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:48.033550 14105 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:48.033598 14105 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.033629 14105 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:48.033653 14105 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724088, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:48.040470 14105 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:44707
I20240503 08:14:48.040598 14235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:44707 every 8 connection(s)
I20240503 08:14:48.040823 14105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:14:48.043079 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.034137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:46317 (local address 127.6.68.254:43581)
0503 08:14:48.034177 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.034179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.034189 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:14:48.034635 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.034637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.034649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.034710 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:48.035649 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.036480 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.037436 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.037665 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.040149 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:48.040154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:48.040155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:48.040160 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:48.041682 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.042221 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.042222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.042223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.042260 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.042484 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.042484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.042485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.042580 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:48.042639 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.042888 (+ 249us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.042985 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240503 08:14:48.043185 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.033855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43581 (local address 127.6.68.193:46317)
0503 08:14:48.033995 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.034024 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.034534 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.034771 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.034776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.035099 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:48.035504 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.035510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.036601 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.036602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.037347 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.037350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.037499 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.038117 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:48.038139 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:48.040054 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:48.041770 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.041777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.041780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.042054 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.042334 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.042335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.042335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.042427 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.042633 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:48.042637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:48.042760 (+ 123us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.042861 (+ 101us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.043003 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20240503 08:14:48.043889 14236 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43581
I20240503 08:14:48.043975 14236 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:48.044171 14236 heartbeater.cc:511] Master 127.6.68.254:43581 requested a full tablet report, sending...
I20240503 08:14:48.044627 14032 ts_manager.cc:194] Registered new tserver with Master: d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707)
I20240503 08:14:48.045264 14032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:46317
I20240503 08:14:48.048012 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 14105
I20240503 08:14:48.048125 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:14:48.048382 6419 external_mini_cluster.cc:1428] Setting key 8ceba6971488b6a9a1a25c6396c8d8a7
I20240503 08:14:48.052006 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.045616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:59383 (local address 127.6.68.254:43581)
0503 08:14:48.045652 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.045654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.045840 (+ 186us) server_negotiation.cc:407] Connection header received
0503 08:14:48.045899 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.045900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.045911 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.045964 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:48.046348 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.046799 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.047294 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.047486 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.048357 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:48.048361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:48.048361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:48.048367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:48.049779 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.051377 (+ 1598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.051378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.051378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.051408 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.051563 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.051563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.051564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.051667 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:48.051728 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.051876 (+ 148us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.051943 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240503 08:14:48.052122 14242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.045585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43581 (local address 127.6.68.193:59383)
0503 08:14:48.045773 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.045783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.045841 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.045969 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.045970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.046111 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:48.046279 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.046280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.046864 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.046865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.047226 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.047227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.047292 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.047916 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:48.047921 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:48.048252 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:48.051067 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.051068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.051069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.051287 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.051486 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.051486 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.051487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.051515 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.051729 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:48.051730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:48.051780 (+ 50us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.051861 (+ 81us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.051976 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab.
Entry for principal oryx/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab.
I20240503 08:14:48.074702 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43581
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab
--principal=oryx/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:44847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:48.152051 14245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:48.152212 14245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:48.152241 14245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:48.153497 14245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:48.153540 14245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:48.153559 14245 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240503 08:14:48.153575 14245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:14:48.155107 14245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:44847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab
--principal=oryx/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.68.254:43581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:48.155292 14245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:48.155464 14245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:48.159945 14253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:48.160534 14250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:48.161180 14251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:48.162075 14245 server_base.cc:1034] running on GCE node
I20240503 08:14:48.162250 14245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:48.162452 14245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:48.163589 14245 hybrid_clock.cc:648] HybridClock initialized: now 1714724088163567 us; error 34 us; skew 500 ppm
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724088, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:48.165891 14245 init.cc:377] Logged in from keytab as oryx/127.6.68.194@KRBTEST.COM (short username oryx)
I20240503 08:14:48.166280 14245 webserver.cc:460] Webserver started at http://127.6.68.194:44759/ using document root <none> and password file <none>
I20240503 08:14:48.166440 14245 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:48.166481 14245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:48.166589 14245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:48.167536 14245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "b7bda12194ac49d29e0b0db730db7169"
format_stamp: "Formatted at 2024-05-03 08:14:48 on dist-test-slave-3g67"
server_key: "f29a4227c2064d6ec5111323b85c9e53"
server_key_iv: "5c68fd0205dbd26d7c43a90396ed903c"
server_key_version: "encryptionkey@0"
I20240503 08:14:48.167867 14245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "b7bda12194ac49d29e0b0db730db7169"
format_stamp: "Formatted at 2024-05-03 08:14:48 on dist-test-slave-3g67"
server_key: "f29a4227c2064d6ec5111323b85c9e53"
server_key_iv: "5c68fd0205dbd26d7c43a90396ed903c"
server_key_version: "encryptionkey@0"
I20240503 08:14:48.169066 14245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:48.169780 14260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.169939 14245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:14:48.170014 14245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "b7bda12194ac49d29e0b0db730db7169"
format_stamp: "Formatted at 2024-05-03 08:14:48 on dist-test-slave-3g67"
server_key: "f29a4227c2064d6ec5111323b85c9e53"
server_key_iv: "5c68fd0205dbd26d7c43a90396ed903c"
server_key_version: "encryptionkey@0"
I20240503 08:14:48.170076 14245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:48.187207 14245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:48.189229 14245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:48.189430 14245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:48.189936 14245 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:48.189975 14245 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.190008 14245 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:48.190039 14245 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724088, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:48.196622 14245 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:38409
I20240503 08:14:48.197010 14245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:14:48.196992 14375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:38409 every 8 connection(s)
I20240503 08:14:48.198334 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.190313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:56713 (local address 127.6.68.254:43581)
0503 08:14:48.190378 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.190381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.190547 (+ 166us) server_negotiation.cc:407] Connection header received
0503 08:14:48.191021 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.191023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.191036 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.191120 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:48.192066 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.192900 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.193690 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.193909 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.195870 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:48.195875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:48.195875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:48.195883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:48.197457 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.197770 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.197771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.197771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.197788 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.197912 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.197912 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.197913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.197990 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:48.198029 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.198214 (+ 185us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.198273 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240503 08:14:48.198436 14269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.190270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43581 (local address 127.6.68.194:56713)
0503 08:14:48.190438 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.190464 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.190925 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.191177 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.191180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.191515 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:48.191953 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.191957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.192992 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.192992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.193588 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.193590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.193722 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.194231 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:48.194250 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:48.195772 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:48.197516 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.197519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.197521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.197713 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.197826 (+ 113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.197827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.197827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.197870 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.198027 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:48.198029 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:48.198122 (+ 93us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.198199 (+ 77us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.198295 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
I20240503 08:14:48.198899 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 14245
I20240503 08:14:48.198992 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:14:48.199177 6419 external_mini_cluster.cc:1428] Setting key d8b0680de82c6744ef3b39099276b479
I20240503 08:14:48.199482 14376 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43581
I20240503 08:14:48.199584 14376 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:48.199965 14376 heartbeater.cc:511] Master 127.6.68.254:43581 requested a full tablet report, sending...
I20240503 08:14:48.200335 14033 ts_manager.cc:194] Registered new tserver with Master: b7bda12194ac49d29e0b0db730db7169 (127.6.68.194:38409)
I20240503 08:14:48.200718 14033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:56713
I20240503 08:14:48.205318 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.199966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43581 (local address 127.6.68.194:52995)
0503 08:14:48.200150 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.200159 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.200221 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.200333 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.200334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.200515 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:48.200755 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.200757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.201789 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.201790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.202144 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.202145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.202219 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.202741 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:48.202756 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:48.202983 (+ 227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:48.204507 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.204508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.204509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.204643 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.204772 (+ 129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.204772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.204773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.204798 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.205009 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:48.205010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:48.205052 (+ 42us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.205116 (+ 64us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.205210 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
I20240503 08:14:48.205332 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.200013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:52995 (local address 127.6.68.254:43581)
0503 08:14:48.200059 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.200061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.200210 (+ 149us) server_negotiation.cc:407] Connection header received
0503 08:14:48.200265 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.200266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.200277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.200328 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:48.200993 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.201687 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.202229 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.202428 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.203057 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:48.203061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:48.203061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:48.203067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:48.204435 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.204707 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.204708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.204708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.204725 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.204851 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.204852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.204852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.204956 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:48.205012 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.205142 (+ 130us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.205214 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for oryx/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab.
Entry for principal oryx/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab.
I20240503 08:14:48.224691 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43581
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab
--principal=oryx/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:44847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:48.300961 14385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:48.301123 14385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:48.301147 14385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:48.302371 14385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:48.302412 14385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:48.302430 14385 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240503 08:14:48.302446 14385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:14:48.303977 14385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:44847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab
--principal=oryx/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.68.254:43581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:48.304147 14385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:48.304289 14385 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:48.308648 14393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:48.309290 14390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:48.310063 14391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:48.310850 14385 server_base.cc:1034] running on GCE node
I20240503 08:14:48.311033 14385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:48.311228 14385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:48.312392 14385 hybrid_clock.cc:648] HybridClock initialized: now 1714724088312372 us; error 36 us; skew 500 ppm
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724088, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:48.314742 14385 init.cc:377] Logged in from keytab as oryx/127.6.68.195@KRBTEST.COM (short username oryx)
I20240503 08:14:48.315140 14385 webserver.cc:460] Webserver started at http://127.6.68.195:43797/ using document root <none> and password file <none>
I20240503 08:14:48.315323 14385 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:48.315366 14385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:48.315479 14385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:48.316471 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "4208988005e8442fa4f538a5ef2b1f5d"
format_stamp: "Formatted at 2024-05-03 08:14:48 on dist-test-slave-3g67"
server_key: "271b826060a531ae63ad651805128db7"
server_key_iv: "79173c941e9349c666d039796b616b6b"
server_key_version: "encryptionkey@0"
I20240503 08:14:48.316756 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "4208988005e8442fa4f538a5ef2b1f5d"
format_stamp: "Formatted at 2024-05-03 08:14:48 on dist-test-slave-3g67"
server_key: "271b826060a531ae63ad651805128db7"
server_key_iv: "79173c941e9349c666d039796b616b6b"
server_key_version: "encryptionkey@0"
I20240503 08:14:48.317900 14385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:14:48.318526 14400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.318681 14385 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:14:48.318753 14385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "4208988005e8442fa4f538a5ef2b1f5d"
format_stamp: "Formatted at 2024-05-03 08:14:48 on dist-test-slave-3g67"
server_key: "271b826060a531ae63ad651805128db7"
server_key_iv: "79173c941e9349c666d039796b616b6b"
server_key_version: "encryptionkey@0"
I20240503 08:14:48.318809 14385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:48.331086 14385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:48.333213 14385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:48.333328 14385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:48.333771 14385 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:48.333817 14385 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.333850 14385 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:48.333882 14385 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724088, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:48.340171 14385 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:43407
I20240503 08:14:48.340234 14515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:43407 every 8 connection(s)
I20240503 08:14:48.340540 14385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:14:48.342492 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.334081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:48489 (local address 127.6.68.254:43581)
0503 08:14:48.334127 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.334129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.334329 (+ 200us) server_negotiation.cc:407] Connection header received
0503 08:14:48.334747 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.334749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.334760 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.334845 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:48.335741 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.336567 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.337352 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.337554 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.340078 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:48.340081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:48.340082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:48.340088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:48.341573 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.341898 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.341899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.341899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.341919 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.342055 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.342055 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.342055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.342138 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:48.342183 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.342373 (+ 190us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.342430 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:14:48.342614 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.334063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43581 (local address 127.6.68.195:48489)
0503 08:14:48.334185 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.334212 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.334658 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.334888 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.334891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.335214 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:48.335645 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.335651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.336660 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.336662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.337267 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.337269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.337411 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.337988 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:48.338007 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:48.339957 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:48.341645 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.341648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.341650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.341835 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.341968 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.341969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.341969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.342013 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.342180 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:48.342183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:48.342278 (+ 95us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.342358 (+ 80us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.342454 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
I20240503 08:14:48.343058 14516 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43581
I20240503 08:14:48.343134 14516 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:48.343353 14516 heartbeater.cc:511] Master 127.6.68.254:43581 requested a full tablet report, sending...
I20240503 08:14:48.343784 14033 ts_manager.cc:194] Registered new tserver with Master: 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407)
I20240503 08:14:48.344215 14033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:48489
I20240503 08:14:48.348477 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.344203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:39505 (local address 127.6.68.254:43581)
0503 08:14:48.344238 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.344240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.344392 (+ 152us) server_negotiation.cc:407] Connection header received
0503 08:14:48.344413 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.344414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.344423 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.344469 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:48.344850 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.345290 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.345776 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.345947 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.346519 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:48.346522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:48.346522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:48.346527 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:48.347602 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.347880 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.347881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.347881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.347898 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.348083 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.348084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.348084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.348150 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:48.348195 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.348351 (+ 156us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.348403 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240503 08:14:48.348536 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.344122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43581 (local address 127.6.68.195:39505)
0503 08:14:48.344232 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.344242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.344327 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.344472 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.344473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.344618 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:48.344783 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.344784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.345355 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.345356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.345704 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.345705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.345765 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.346226 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:48.346230 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:48.346452 (+ 222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:48.347688 (+ 1236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.347689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.347689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.347825 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.347997 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.347998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.347998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.348027 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.348222 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:48.348223 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:48.348262 (+ 39us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.348336 (+ 74us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.348412 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20240503 08:14:48.348750 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 14385
I20240503 08:14:48.348842 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:14:48.349037 6419 external_mini_cluster.cc:1428] Setting key 0d31a84a4a8f1b8449874f322f38a79d
I20240503 08:14:48.350042 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724088, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
W20240503 08:14:48.368835 14102 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:48.363942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.6.68.254:43581)
0503 08:14:48.363997 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.363999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.364008 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:14:48.364071 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.364072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.364080 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.364129 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:48.364848 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.365317 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.365894 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.365992 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.368751 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52560: BlockingRecv error: recv got EOF from 127.0.0.1:52560 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724088, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:48.375485 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.369541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.6.68.254:43581)
0503 08:14:48.369588 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.369590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.369657 (+ 67us) server_negotiation.cc:407] Connection header received
0503 08:14:48.369719 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.369719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.369726 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.369774 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:48.370266 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.370708 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.371173 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.371270 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.373131 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:48.373134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:48.373134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:48.373140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:48.374740 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.375024 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.375025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.375025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.375041 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.375133 (+ 92us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.375134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.375134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.375208 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:48.375278 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.375365 (+ 87us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.375421 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240503 08:14:48.377012 14033 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:14:48.378479 14033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:14:48.383291 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.380378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38409 (local address 127.0.0.1:59090)
0503 08:14:48.380481 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.380507 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.380587 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.380999 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.381002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.381005 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:48.381262 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.381264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.382213 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.382214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.382974 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.382975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.383033 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.383069 (+ 36us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.383120 (+ 51us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.383158 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":47,"threads_started":1}
I20240503 08:14:48.383783 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.380477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.6.68.194:38409)
0503 08:14:48.380833 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.380840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.380848 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:14:48.380880 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.380881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.380893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.380997 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:48.381341 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.382134 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.383067 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.383579 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.383614 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.383685 (+ 71us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.383715 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":93,"threads_started":1}
I20240503 08:14:48.384255 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.380543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:44707 (local address 127.0.0.1:44162)
0503 08:14:48.380658 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.380667 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.380705 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.381098 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.381099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.381101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:48.381275 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.381277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.382223 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.382224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.383005 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.383007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.384089 (+ 1082us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.384092 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.384156 (+ 64us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.384193 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":80,"threads_started":1}
I20240503 08:14:48.384305 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.380646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.6.68.193:44707)
0503 08:14:48.380854 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.380863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.380873 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:14:48.380922 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.380924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.380940 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.381018 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:48.381375 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.382147 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.383124 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.383639 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.383684 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.384181 (+ 497us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.384233 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20240503 08:14:48.384371 14310 tablet_service.cc:1467] Processing CreateTablet for tablet 6fda02d169ea40939709c3ebc4afcb63 (DEFAULT_TABLE table=test-table [id=09ac557aa93e4bf4827c262774a9d350]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:48.384640 14310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fda02d169ea40939709c3ebc4afcb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.384784 14152 tablet_service.cc:1467] Processing CreateTablet for tablet 6fda02d169ea40939709c3ebc4afcb63 (DEFAULT_TABLE table=test-table [id=09ac557aa93e4bf4827c262774a9d350]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:48.384984 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.380714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:43407 (local address 127.0.0.1:36358)
0503 08:14:48.382255 (+ 1541us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.382264 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.382324 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.382482 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.382483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.382485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:48.382678 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.382679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.383952 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.383953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.384792 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.384794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.384855 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.384860 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.384893 (+ 33us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.384924 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1523,"thread_start_us":81,"threads_started":1}
I20240503 08:14:48.385021 14152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fda02d169ea40939709c3ebc4afcb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.385458 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.380803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.6.68.195:43407)
0503 08:14:48.380986 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.380996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.382325 (+ 1329us) server_negotiation.cc:407] Connection header received
0503 08:14:48.382387 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.382389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.382405 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.382482 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:48.382748 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.383871 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.384890 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.385322 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.385350 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.385375 (+ 25us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.385407 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20240503 08:14:48.385849 14450 tablet_service.cc:1467] Processing CreateTablet for tablet 6fda02d169ea40939709c3ebc4afcb63 (DEFAULT_TABLE table=test-table [id=09ac557aa93e4bf4827c262774a9d350]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:48.386037 14450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fda02d169ea40939709c3ebc4afcb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.387663 14544 tablet_bootstrap.cc:492] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: Bootstrap starting.
I20240503 08:14:48.388242 14546 tablet_bootstrap.cc:492] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d: Bootstrap starting.
I20240503 08:14:48.388242 14545 tablet_bootstrap.cc:492] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4: Bootstrap starting.
I20240503 08:14:48.388726 14544 tablet_bootstrap.cc:654] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.388998 14544 log.cc:824] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:48.389148 14546 tablet_bootstrap.cc:654] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.389148 14545 tablet_bootstrap.cc:654] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.389461 14546 log.cc:824] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:48.389474 14545 log.cc:824] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:48.389686 14544 tablet_bootstrap.cc:492] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: No bootstrap required, opened a new log
I20240503 08:14:48.389758 14544 ts_tablet_manager.cc:1367] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:48.390087 14546 tablet_bootstrap.cc:492] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d: No bootstrap required, opened a new log
I20240503 08:14:48.390143 14546 ts_tablet_manager.cc:1367] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:48.390507 14545 tablet_bootstrap.cc:492] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4: No bootstrap required, opened a new log
I20240503 08:14:48.390614 14545 ts_tablet_manager.cc:1367] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:48.391124 14544 raft_consensus.cc:348] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.391278 14544 raft_consensus.cc:374] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.391320 14544 raft_consensus.cc:729] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bda12194ac49d29e0b0db730db7169, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.391417 14544 consensus_queue.cc:260] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.391659 14544 ts_tablet_manager.cc:1398] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:48.391788 14376 heartbeater.cc:503] Master 127.6.68.254:43581 was elected leader, sending a full tablet report...
I20240503 08:14:48.391965 14546 raft_consensus.cc:348] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.392098 14546 raft_consensus.cc:374] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.392129 14546 raft_consensus.cc:729] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4208988005e8442fa4f538a5ef2b1f5d, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.392223 14546 consensus_queue.cc:260] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.392452 14546 ts_tablet_manager.cc:1398] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:48.392606 14516 heartbeater.cc:503] Master 127.6.68.254:43581 was elected leader, sending a full tablet report...
I20240503 08:14:48.392567 14545 raft_consensus.cc:348] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.392658 14545 raft_consensus.cc:374] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.392674 14545 raft_consensus.cc:729] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4be1c81b23d41d59b3af62afdf5c4a4, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.392743 14545 consensus_queue.cc:260] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.392957 14545 ts_tablet_manager.cc:1398] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:14:48.392994 14236 heartbeater.cc:503] Master 127.6.68.254:43581 was elected leader, sending a full tablet report...
I20240503 08:14:48.423322 14552 raft_consensus.cc:482] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:48.423439 14552 raft_consensus.cc:504] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.423828 14552 leader_election.cc:288] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7bda12194ac49d29e0b0db730db7169 (127.6.68.194:38409), 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407)
I20240503 08:14:48.426146 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.424009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38409 (local address 127.6.68.193:35291)
0503 08:14:48.424072 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.424087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.424144 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.424338 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.424339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.424341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:48.424576 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.424578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.425308 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.425309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.425947 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.425948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.426008 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.426012 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.426035 (+ 23us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.426062 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20240503 08:14:48.426290 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.424114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:43407 (local address 127.6.68.193:34831)
0503 08:14:48.424286 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.424295 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.424347 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.424528 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.424529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.424531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:48.424781 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.424782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.425496 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.425497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.426053 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.426054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.426128 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.426132 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.426164 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.426199 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":118,"threads_started":1}
I20240503 08:14:48.426613 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.424096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:35291 (local address 127.6.68.194:38409)
0503 08:14:48.424140 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.424148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.424163 (+ 15us) server_negotiation.cc:407] Connection header received
0503 08:14:48.424220 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.424221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.424231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.424324 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:48.424677 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.425231 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.426027 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.426507 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.426516 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.426539 (+ 23us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.426569 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240503 08:14:48.426780 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.424249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:34831 (local address 127.6.68.195:43407)
0503 08:14:48.424295 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.424298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.424340 (+ 42us) server_negotiation.cc:407] Connection header received
0503 08:14:48.424400 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.424401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.424412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.424474 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:48.424864 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.425407 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.426141 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.426623 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.426629 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.426647 (+ 18us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.426680 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:14:48.426927 14330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fda02d169ea40939709c3ebc4afcb63" candidate_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7bda12194ac49d29e0b0db730db7169" is_pre_election: true
I20240503 08:14:48.427081 14330 raft_consensus.cc:2454] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4be1c81b23d41d59b3af62afdf5c4a4 in term 0.
I20240503 08:14:48.427057 14460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fda02d169ea40939709c3ebc4afcb63" candidate_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4208988005e8442fa4f538a5ef2b1f5d" is_pre_election: true
I20240503 08:14:48.427196 14460 raft_consensus.cc:2454] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4be1c81b23d41d59b3af62afdf5c4a4 in term 0.
I20240503 08:14:48.427306 14123 leader_election.cc:302] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7bda12194ac49d29e0b0db730db7169, d4be1c81b23d41d59b3af62afdf5c4a4; no voters:
I20240503 08:14:48.427450 14552 raft_consensus.cc:2781] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:48.427527 14552 raft_consensus.cc:482] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:48.427553 14552 raft_consensus.cc:3037] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.428323 14552 raft_consensus.cc:504] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.428520 14552 leader_election.cc:288] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [CANDIDATE]: Term 1 election: Requested vote from peers b7bda12194ac49d29e0b0db730db7169 (127.6.68.194:38409), 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407)
I20240503 08:14:48.428584 14330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fda02d169ea40939709c3ebc4afcb63" candidate_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7bda12194ac49d29e0b0db730db7169"
I20240503 08:14:48.428642 14330 raft_consensus.cc:3037] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.428658 14460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fda02d169ea40939709c3ebc4afcb63" candidate_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4208988005e8442fa4f538a5ef2b1f5d"
I20240503 08:14:48.428720 14460 raft_consensus.cc:3037] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.429388 14330 raft_consensus.cc:2454] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4be1c81b23d41d59b3af62afdf5c4a4 in term 1.
I20240503 08:14:48.429467 14460 raft_consensus.cc:2454] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4be1c81b23d41d59b3af62afdf5c4a4 in term 1.
I20240503 08:14:48.429581 14123 leader_election.cc:302] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7bda12194ac49d29e0b0db730db7169, d4be1c81b23d41d59b3af62afdf5c4a4; no voters:
I20240503 08:14:48.429684 14552 raft_consensus.cc:2781] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:48.429845 14552 raft_consensus.cc:686] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 LEADER]: Becoming Leader. State: Replica: d4be1c81b23d41d59b3af62afdf5c4a4, State: Running, Role: LEADER
I20240503 08:14:48.429926 14552 consensus_queue.cc:237] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.430676 14033 catalog_manager.cc:5484] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193). New cstate: current_term: 1 leader_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:48.444459 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.442328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.6.68.193:44707)
0503 08:14:48.442404 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.442406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.442415 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:14:48.442451 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.442451 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.442467 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.442573 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:14:48.442848 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.443359 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.443993 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.444133 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.444169 (+ 36us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:14:48.444254 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:14:48.444321 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.444359 (+ 38us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.444381 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240503 08:14:48.446111 14460 raft_consensus.cc:1261] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Refusing update from remote peer d4be1c81b23d41d59b3af62afdf5c4a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:48.446111 14330 raft_consensus.cc:1261] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Refusing update from remote peer d4be1c81b23d41d59b3af62afdf5c4a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:48.446651 14554 consensus_queue.cc:1035] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:48.446799 14552 consensus_queue.cc:1035] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:14:48.447553 14377 tablet.cc:2369] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:48.449065 14558 mvcc.cc:205] Tried to move back new op lower bound from 7023509866272702464 to 7023509866209509376. Current Snapshot: MvccSnapshot[applied={T|T < 7023509866272702464}]
I20240503 08:14:48.449376 14557 mvcc.cc:205] Tried to move back new op lower bound from 7023509866272702464 to 7023509866209509376. Current Snapshot: MvccSnapshot[applied={T|T < 7023509866272702464 or (T in {7023509866272702464})}]
I20240503 08:14:48.451192 14033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52570:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:14:48.451294 14033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52570:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:14:48.451983 14033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744: Sending DeleteTablet for 3 replicas of tablet 6fda02d169ea40939709c3ebc4afcb63
I20240503 08:14:48.452322 14310 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fda02d169ea40939709c3ebc4afcb63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:48 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:59090
I20240503 08:14:48.452380 14450 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fda02d169ea40939709c3ebc4afcb63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:48 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:36358
I20240503 08:14:48.452585 14567 tablet_replica.cc:330] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d: stopping tablet replica
I20240503 08:14:48.452687 14567 raft_consensus.cc:2229] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:48.452669 14152 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fda02d169ea40939709c3ebc4afcb63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:48 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:44162
I20240503 08:14:48.452807 14567 raft_consensus.cc:2258] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:48.452832 14033 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 08:14:48.452915 14568 tablet_replica.cc:330] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4: stopping tablet replica
W20240503 08:14:48.452973 14033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:14:48.452989 14568 raft_consensus.cc:2229] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:14:48.453068 14568 raft_consensus.cc:2258] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:48.453663 14568 ts_tablet_manager.cc:1875] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:48.454285 14152 tablet_service.cc:1467] Processing CreateTablet for tablet c20646678969482994075f88801d101f (DEFAULT_TABLE table=test-table [id=0b86f901f1414dfeac242df4f9411f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:48.454290 14310 tablet_service.cc:1467] Processing CreateTablet for tablet c20646678969482994075f88801d101f (DEFAULT_TABLE table=test-table [id=0b86f901f1414dfeac242df4f9411f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:48.454403 14152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20646678969482994075f88801d101f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.454406 14310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20646678969482994075f88801d101f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.455276 14544 tablet_bootstrap.cc:492] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169: Bootstrap starting.
I20240503 08:14:48.455682 14545 tablet_bootstrap.cc:492] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4: Bootstrap starting.
I20240503 08:14:48.455693 14544 tablet_bootstrap.cc:654] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.455720 14450 tablet_service.cc:1467] Processing CreateTablet for tablet c20646678969482994075f88801d101f (DEFAULT_TABLE table=test-table [id=0b86f901f1414dfeac242df4f9411f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:48.455821 14450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20646678969482994075f88801d101f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.456066 14545 tablet_bootstrap.cc:654] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.456588 14568 ts_tablet_manager.cc:1888] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:14:48.456652 14568 log.cc:1196] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/6fda02d169ea40939709c3ebc4afcb63
I20240503 08:14:48.456717 14545 tablet_bootstrap.cc:492] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4: No bootstrap required, opened a new log
I20240503 08:14:48.456754 14545 ts_tablet_manager.cc:1367] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:48.456916 14568 ts_tablet_manager.cc:1909] T 6fda02d169ea40939709c3ebc4afcb63 P d4be1c81b23d41d59b3af62afdf5c4a4: Deleting consensus metadata
I20240503 08:14:48.456900 14545 raft_consensus.cc:348] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.456980 14545 raft_consensus.cc:374] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.457006 14545 raft_consensus.cc:729] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4be1c81b23d41d59b3af62afdf5c4a4, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.457042 14545 consensus_queue.cc:260] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.457132 14545 ts_tablet_manager.cc:1398] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:14:48.457213 14546 tablet_bootstrap.cc:492] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d: Bootstrap starting.
I20240503 08:14:48.457458 14567 ts_tablet_manager.cc:1875] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:48.457506 14546 tablet_bootstrap.cc:654] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.458199 14546 tablet_bootstrap.cc:492] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d: No bootstrap required, opened a new log
I20240503 08:14:48.458199 14019 catalog_manager.cc:4830] TS d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707): tablet 6fda02d169ea40939709c3ebc4afcb63 (table test-table [id=09ac557aa93e4bf4827c262774a9d350]) successfully deleted
I20240503 08:14:48.458250 14546 ts_tablet_manager.cc:1367] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:48.458384 14546 raft_consensus.cc:348] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.458443 14546 raft_consensus.cc:374] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.458467 14546 raft_consensus.cc:729] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4208988005e8442fa4f538a5ef2b1f5d, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.458518 14546 consensus_queue.cc:260] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.458588 14546 ts_tablet_manager.cc:1398] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.459816 14567 ts_tablet_manager.cc:1888] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:14:48.459887 14567 log.cc:1196] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/6fda02d169ea40939709c3ebc4afcb63
I20240503 08:14:48.459905 14566 tablet_replica.cc:330] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: stopping tablet replica
I20240503 08:14:48.459988 14566 raft_consensus.cc:2229] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:48.460023 14544 tablet_bootstrap.cc:492] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169: No bootstrap required, opened a new log
I20240503 08:14:48.460075 14544 ts_tablet_manager.cc:1367] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:14:48.460089 14566 raft_consensus.cc:2258] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:48.460163 14567 ts_tablet_manager.cc:1909] T 6fda02d169ea40939709c3ebc4afcb63 P 4208988005e8442fa4f538a5ef2b1f5d: Deleting consensus metadata
I20240503 08:14:48.460253 14544 raft_consensus.cc:348] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.460350 14544 raft_consensus.cc:374] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.460562 14544 raft_consensus.cc:729] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bda12194ac49d29e0b0db730db7169, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.460614 14544 consensus_queue.cc:260] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.460635 14019 catalog_manager.cc:4830] TS 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407): tablet 6fda02d169ea40939709c3ebc4afcb63 (table test-table [id=09ac557aa93e4bf4827c262774a9d350]) successfully deleted
I20240503 08:14:48.460656 14566 ts_tablet_manager.cc:1875] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:48.460712 14544 ts_tablet_manager.cc:1398] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:14:48.461908 14566 ts_tablet_manager.cc:1888] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:14:48.461964 14566 log.cc:1196] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/6fda02d169ea40939709c3ebc4afcb63
I20240503 08:14:48.462174 14566 ts_tablet_manager.cc:1909] T 6fda02d169ea40939709c3ebc4afcb63 P b7bda12194ac49d29e0b0db730db7169: Deleting consensus metadata
I20240503 08:14:48.462519 14019 catalog_manager.cc:4830] TS b7bda12194ac49d29e0b0db730db7169 (127.6.68.194:38409): tablet 6fda02d169ea40939709c3ebc4afcb63 (table test-table [id=09ac557aa93e4bf4827c262774a9d350]) successfully deleted
W20240503 08:14:48.542382 14237 tablet.cc:2369] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:14:48.591359 14517 tablet.cc:2369] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:14:48.692466 14550 raft_consensus.cc:482] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:48.692580 14550 raft_consensus.cc:504] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.692924 14550 leader_election.cc:288] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707), 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407)
I20240503 08:14:48.695021 14269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.693111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:44707 (local address 127.6.68.194:49605)
0503 08:14:48.693197 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.693220 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.693270 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.693453 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.693454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.693457 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:48.693630 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.693631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.694230 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.694231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.694805 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.694806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.694877 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.694881 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.694914 (+ 33us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.694952 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20240503 08:14:48.695147 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.693272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:43407 (local address 127.6.68.194:56049)
0503 08:14:48.693376 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.693384 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.693438 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.693574 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.693575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.693577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:48.693771 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.693772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.694365 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.694366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.694930 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.694931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.694999 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.695002 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.695034 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.695068 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20240503 08:14:48.695485 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.693190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:49605 (local address 127.6.68.193:44707)
0503 08:14:48.693242 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.693244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.693257 (+ 13us) server_negotiation.cc:407] Connection header received
0503 08:14:48.693347 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.693348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.693358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.693444 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:48.693697 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.694160 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.694890 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.695383 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.695392 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.695415 (+ 23us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.695439 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240503 08:14:48.695541 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.693363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:56049 (local address 127.6.68.195:43407)
0503 08:14:48.693417 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.693419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.693429 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:14:48.693502 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.693503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.693513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.693560 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:48.693835 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.694288 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.695005 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.695446 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.695453 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.695472 (+ 19us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.695495 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240503 08:14:48.695746 14460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20646678969482994075f88801d101f" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4208988005e8442fa4f538a5ef2b1f5d" is_pre_election: true
I20240503 08:14:48.695755 14180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20646678969482994075f88801d101f" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" is_pre_election: true
I20240503 08:14:48.695843 14460 raft_consensus.cc:2454] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 0.
I20240503 08:14:48.695858 14180 raft_consensus.cc:2454] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 0.
I20240503 08:14:48.696035 14263 leader_election.cc:302] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7bda12194ac49d29e0b0db730db7169, d4be1c81b23d41d59b3af62afdf5c4a4; no voters:
I20240503 08:14:48.696170 14550 raft_consensus.cc:2781] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:48.696230 14550 raft_consensus.cc:482] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:48.696298 14550 raft_consensus.cc:3037] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.696946 14550 raft_consensus.cc:504] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.697062 14550 leader_election.cc:288] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 election: Requested vote from peers d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707), 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407)
I20240503 08:14:48.697248 14180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20646678969482994075f88801d101f" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4"
I20240503 08:14:48.697259 14460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20646678969482994075f88801d101f" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4208988005e8442fa4f538a5ef2b1f5d"
I20240503 08:14:48.697330 14460 raft_consensus.cc:3037] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.697330 14180 raft_consensus.cc:3037] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.698112 14460 raft_consensus.cc:2454] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 1.
I20240503 08:14:48.698252 14180 raft_consensus.cc:2454] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 1.
I20240503 08:14:48.698321 14263 leader_election.cc:302] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4208988005e8442fa4f538a5ef2b1f5d, b7bda12194ac49d29e0b0db730db7169; no voters:
I20240503 08:14:48.698423 14550 raft_consensus.cc:2781] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:48.698531 14550 raft_consensus.cc:686] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 1 LEADER]: Becoming Leader. State: Replica: b7bda12194ac49d29e0b0db730db7169, State: Running, Role: LEADER
I20240503 08:14:48.698596 14550 consensus_queue.cc:237] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.699278 14031 catalog_manager.cc:5484] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7bda12194ac49d29e0b0db730db7169 (127.6.68.194). New cstate: current_term: 1 leader_uuid: "b7bda12194ac49d29e0b0db730db7169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } health_report { overall_health: UNKNOWN } } }
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724087, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.254@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:48.763100 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.756499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43581 (local address 127.0.0.1:52584)
0503 08:14:48.756630 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.756639 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.756722 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.756841 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.756842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.757170 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:48.757374 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.757376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.758034 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.758035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.758431 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.758432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.758496 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.759025 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:48.759029 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:48.760508 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:48.762314 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.762320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.762323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.762495 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.762622 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:48.762622 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:48.762623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:48.762646 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:48.762817 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:48.762822 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:48.762860 (+ 38us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.762914 (+ 54us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.763003 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20240503 08:14:48.763144 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.756553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52584 (local address 127.6.68.254:43581)
0503 08:14:48.756648 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.756649 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.756718 (+ 69us) server_negotiation.cc:407] Connection header received
0503 08:14:48.756772 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.756773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.756781 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.756831 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:48.757462 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.757951 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.758524 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.758650 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.760610 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:48.760614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:48.760614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:48.760620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:48.762190 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.762559 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.762560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.762560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.762577 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.762685 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.762685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.762686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.762766 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:48.762815 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.762947 (+ 132us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.763047 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":71}
I20240503 08:14:48.765089 14031 catalog_manager.cc:1904] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:52584:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240503 08:14:48.765272 14031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:14:48.766736 14152 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb39fc47f804d3b9e5620428b4cc325 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94331f71f8f9496da5bfaeb98c7ff2e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240503 08:14:48.766778 14310 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb39fc47f804d3b9e5620428b4cc325 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94331f71f8f9496da5bfaeb98c7ff2e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240503 08:14:48.766808 14450 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb39fc47f804d3b9e5620428b4cc325 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94331f71f8f9496da5bfaeb98c7ff2e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240503 08:14:48.766892 14310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb39fc47f804d3b9e5620428b4cc325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.766892 14152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb39fc47f804d3b9e5620428b4cc325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.766954 14450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb39fc47f804d3b9e5620428b4cc325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.768177 14546 tablet_bootstrap.cc:492] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d: Bootstrap starting.
I20240503 08:14:48.768316 14544 tablet_bootstrap.cc:492] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: Bootstrap starting.
I20240503 08:14:48.768317 14545 tablet_bootstrap.cc:492] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4: Bootstrap starting.
I20240503 08:14:48.768616 14544 tablet_bootstrap.cc:654] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.768636 14545 tablet_bootstrap.cc:654] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.768633 14546 tablet_bootstrap.cc:654] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.769225 14544 tablet_bootstrap.cc:492] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: No bootstrap required, opened a new log
I20240503 08:14:48.769287 14544 ts_tablet_manager.cc:1367] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:14:48.769394 14546 tablet_bootstrap.cc:492] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d: No bootstrap required, opened a new log
I20240503 08:14:48.769393 14545 tablet_bootstrap.cc:492] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4: No bootstrap required, opened a new log
I20240503 08:14:48.769413 14544 raft_consensus.cc:348] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.769434 14545 ts_tablet_manager.cc:1367] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:48.769434 14546 ts_tablet_manager.cc:1367] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:48.769459 14544 raft_consensus.cc:374] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.769471 14544 raft_consensus.cc:729] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bda12194ac49d29e0b0db730db7169, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.769508 14544 consensus_queue.cc:260] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.769553 14545 raft_consensus.cc:348] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.769595 14544 ts_tablet_manager.cc:1398] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.769572 14546 raft_consensus.cc:348] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.769613 14545 raft_consensus.cc:374] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.769629 14546 raft_consensus.cc:374] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.769644 14545 raft_consensus.cc:729] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4be1c81b23d41d59b3af62afdf5c4a4, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.769652 14546 raft_consensus.cc:729] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4208988005e8442fa4f538a5ef2b1f5d, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.769621 14550 tablet_replica.cc:439] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } } }
I20240503 08:14:48.769675 14550 tablet_replica.cc:442] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:14:48.769696 14545 consensus_queue.cc:260] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.769693 14546 consensus_queue.cc:260] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.769783 14546 ts_tablet_manager.cc:1398] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.769786 14545 ts_tablet_manager.cc:1398] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.769814 14552 tablet_replica.cc:439] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } } }
I20240503 08:14:48.769893 14552 tablet_replica.cc:442] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:14:48.770006 14551 tablet_replica.cc:439] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } } }
I20240503 08:14:48.770114 14551 tablet_replica.cc:442] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:14:48.780889 14550 raft_consensus.cc:482] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:48.780978 14550 raft_consensus.cc:504] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.781123 14550 leader_election.cc:288] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707), 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407)
I20240503 08:14:48.781297 14180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb39fc47f804d3b9e5620428b4cc325" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" is_pre_election: true
I20240503 08:14:48.781390 14180 raft_consensus.cc:2454] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 0.
I20240503 08:14:48.781345 14460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb39fc47f804d3b9e5620428b4cc325" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4208988005e8442fa4f538a5ef2b1f5d" is_pre_election: true
I20240503 08:14:48.781440 14460 raft_consensus.cc:2454] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 0.
I20240503 08:14:48.781543 14263 leader_election.cc:302] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7bda12194ac49d29e0b0db730db7169, d4be1c81b23d41d59b3af62afdf5c4a4; no voters:
I20240503 08:14:48.781641 14550 raft_consensus.cc:2781] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:48.781684 14550 raft_consensus.cc:482] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:48.781708 14550 raft_consensus.cc:3037] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.782346 14550 raft_consensus.cc:504] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.782461 14550 leader_election.cc:288] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 election: Requested vote from peers d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707), 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407)
I20240503 08:14:48.782682 14460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb39fc47f804d3b9e5620428b4cc325" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4208988005e8442fa4f538a5ef2b1f5d"
I20240503 08:14:48.782691 14180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb39fc47f804d3b9e5620428b4cc325" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4"
I20240503 08:14:48.782747 14460 raft_consensus.cc:3037] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.782758 14180 raft_consensus.cc:3037] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.783258 14460 raft_consensus.cc:2454] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 1.
I20240503 08:14:48.783334 14180 raft_consensus.cc:2454] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 1.
I20240503 08:14:48.783450 14263 leader_election.cc:302] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4208988005e8442fa4f538a5ef2b1f5d, b7bda12194ac49d29e0b0db730db7169; no voters:
I20240503 08:14:48.783542 14550 raft_consensus.cc:2781] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:48.783601 14550 raft_consensus.cc:686] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [term 1 LEADER]: Becoming Leader. State: Replica: b7bda12194ac49d29e0b0db730db7169, State: Running, Role: LEADER
I20240503 08:14:48.783675 14550 consensus_queue.cc:237] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.783991 14573 tablet_replica.cc:439] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: TxnStatusTablet state changed. Reason: New leader b7bda12194ac49d29e0b0db730db7169. Latest consensus state: current_term: 1 leader_uuid: "b7bda12194ac49d29e0b0db730db7169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } } }
I20240503 08:14:48.784085 14573 tablet_replica.cc:442] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: This TxnStatusTablet replica's current role is: LEADER
I20240503 08:14:48.784269 14585 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240503 08:14:48.784291 14032 catalog_manager.cc:5484] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7bda12194ac49d29e0b0db730db7169 (127.6.68.194). New cstate: current_term: 1 leader_uuid: "b7bda12194ac49d29e0b0db730db7169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:48.784390 14585 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240503 08:14:48.789901 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.787345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38409 (local address 127.0.0.1:59102)
0503 08:14:48.787478 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.787488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.787537 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.787746 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.787746 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.787748 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0503 08:14:48.787889 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.787891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.788780 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.788781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.789209 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.789211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.789280 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.789296 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0503 08:14:48.789749 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0503 08:14:48.789756 (+ 7us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.789800 (+ 44us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.789842 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20240503 08:14:48.789907 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.787416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.6.68.194:38409)
0503 08:14:48.787474 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.787477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.787530 (+ 53us) server_negotiation.cc:407] Connection header received
0503 08:14:48.787611 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.787612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.787659 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.787731 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:14:48.787954 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.788671 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.789294 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.789505 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.789547 (+ 42us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:14:48.789670 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:14:48.789750 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.789815 (+ 65us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.789837 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:14:48.790961 14460 raft_consensus.cc:1261] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Refusing update from remote peer b7bda12194ac49d29e0b0db730db7169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:48.790980 14180 raft_consensus.cc:1261] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Refusing update from remote peer b7bda12194ac49d29e0b0db730db7169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:48.791247 14550 consensus_queue.cc:1035] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:48.791349 14573 consensus_queue.cc:1035] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:48.792012 14551 tablet_replica.cc:439] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d: TxnStatusTablet state changed. Reason: New leader b7bda12194ac49d29e0b0db730db7169. Latest consensus state: current_term: 1 leader_uuid: "b7bda12194ac49d29e0b0db730db7169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } } }
I20240503 08:14:48.792130 14551 tablet_replica.cc:442] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:14:48.792066 14552 tablet_replica.cc:439] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4: TxnStatusTablet state changed. Reason: New leader b7bda12194ac49d29e0b0db730db7169. Latest consensus state: current_term: 1 leader_uuid: "b7bda12194ac49d29e0b0db730db7169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } } }
I20240503 08:14:48.792160 14552 tablet_replica.cc:442] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:14:48.793054 14573 tablet_replica.cc:439] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7bda12194ac49d29e0b0db730db7169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } } }
I20240503 08:14:48.793160 14573 tablet_replica.cc:442] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: This TxnStatusTablet replica's current role is: LEADER
I20240503 08:14:48.793232 14559 mvcc.cc:205] Tried to move back new op lower bound from 7023509867685785600 to 7023509867658276864. Current Snapshot: MvccSnapshot[applied={T|T < 7023509867685785600 or (T in {7023509867685785600})}]
I20240503 08:14:48.793560 14551 tablet_replica.cc:439] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7bda12194ac49d29e0b0db730db7169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } } }
I20240503 08:14:48.793641 14551 tablet_replica.cc:442] T 5eb39fc47f804d3b9e5620428b4cc325 P 4208988005e8442fa4f538a5ef2b1f5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:14:48.794538 14550 tablet_replica.cc:439] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7bda12194ac49d29e0b0db730db7169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } } }
I20240503 08:14:48.794613 14550 tablet_replica.cc:442] T 5eb39fc47f804d3b9e5620428b4cc325 P b7bda12194ac49d29e0b0db730db7169: This TxnStatusTablet replica's current role is: LEADER
I20240503 08:14:48.795095 14552 tablet_replica.cc:439] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7bda12194ac49d29e0b0db730db7169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } } }
I20240503 08:14:48.795171 14552 tablet_replica.cc:442] T 5eb39fc47f804d3b9e5620428b4cc325 P d4be1c81b23d41d59b3af62afdf5c4a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240503 08:14:48.797042 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.795020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.6.68.194:38409)
0503 08:14:48.795087 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.795089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.795097 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:14:48.795116 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.795116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.795125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.795178 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:14:48.795510 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.796047 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.796667 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.796799 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.796816 (+ 17us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:14:48.796835 (+ 19us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:14:48.796885 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.796950 (+ 65us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.796971 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240503 08:14:48.801494 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.799328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:40469 (local address 127.6.68.194:38409)
0503 08:14:48.799371 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.799373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.799481 (+ 108us) server_negotiation.cc:407] Connection header received
0503 08:14:48.799560 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.799566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.799575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.799641 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:14:48.799972 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.800417 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.801078 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.801223 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.801240 (+ 17us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:14:48.801257 (+ 17us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:14:48.801308 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.801380 (+ 72us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.801397 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:14:48.801507 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.799256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38409 (local address 127.6.68.194:40469)
0503 08:14:48.799412 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.799421 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.799494 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.799646 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.799647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.799648 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0503 08:14:48.799887 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.799888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.800519 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.800520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.800996 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.800997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.801079 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.801095 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0503 08:14:48.801314 (+ 219us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0503 08:14:48.801317 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.801361 (+ 44us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.801404 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20240503 08:14:48.803792 14460 raft_consensus.cc:1261] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Refusing update from remote peer b7bda12194ac49d29e0b0db730db7169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:48.803963 14180 raft_consensus.cc:1261] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Refusing update from remote peer b7bda12194ac49d29e0b0db730db7169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:48.803979 14573 consensus_queue.cc:1035] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:48.804201 14573 consensus_queue.cc:1035] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:48.815338 14032 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52570:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:14:48.815433 14032 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52570:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:14:48.816013 14032 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744: Sending DeleteTablet for 3 replicas of tablet c20646678969482994075f88801d101f
I20240503 08:14:48.816259 14152 tablet_service.cc:1514] Processing DeleteTablet for tablet c20646678969482994075f88801d101f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:48 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:44162
I20240503 08:14:48.816326 14309 tablet_service.cc:1514] Processing DeleteTablet for tablet c20646678969482994075f88801d101f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:48 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:59090
I20240503 08:14:48.816375 14568 tablet_replica.cc:330] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4: stopping tablet replica
I20240503 08:14:48.816418 14566 tablet_replica.cc:330] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169: stopping tablet replica
I20240503 08:14:48.816438 14568 raft_consensus.cc:2229] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:48.816466 14566 raft_consensus.cc:2229] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:14:48.816474 14568 raft_consensus.cc:2258] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:48.816535 14566 raft_consensus.cc:2258] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:48.816606 14450 tablet_service.cc:1514] Processing DeleteTablet for tablet c20646678969482994075f88801d101f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:48 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:36358
I20240503 08:14:48.816695 14567 tablet_replica.cc:330] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d: stopping tablet replica
I20240503 08:14:48.816733 14567 raft_consensus.cc:2229] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:48.816773 14567 raft_consensus.cc:2258] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:48.816864 14566 ts_tablet_manager.cc:1875] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:48.817039 14567 ts_tablet_manager.cc:1875] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:48.818935 14566 ts_tablet_manager.cc:1888] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:14:48.818969 14567 ts_tablet_manager.cc:1888] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:14:48.819015 14566 log.cc:1196] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/c20646678969482994075f88801d101f
I20240503 08:14:48.819046 14567 log.cc:1196] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/c20646678969482994075f88801d101f
I20240503 08:14:48.819258 14566 ts_tablet_manager.cc:1909] T c20646678969482994075f88801d101f P b7bda12194ac49d29e0b0db730db7169: Deleting consensus metadata
I20240503 08:14:48.819312 14567 ts_tablet_manager.cc:1909] T c20646678969482994075f88801d101f P 4208988005e8442fa4f538a5ef2b1f5d: Deleting consensus metadata
I20240503 08:14:48.819698 14019 catalog_manager.cc:4830] TS 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407): tablet c20646678969482994075f88801d101f (table test-table [id=0b86f901f1414dfeac242df4f9411f12]) successfully deleted
I20240503 08:14:48.819869 14019 catalog_manager.cc:4830] TS b7bda12194ac49d29e0b0db730db7169 (127.6.68.194:38409): tablet c20646678969482994075f88801d101f (table test-table [id=0b86f901f1414dfeac242df4f9411f12]) successfully deleted
I20240503 08:14:48.820070 14568 ts_tablet_manager.cc:1875] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:48.821625 14568 ts_tablet_manager.cc:1888] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:14:48.821686 14568 log.cc:1196] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/c20646678969482994075f88801d101f
I20240503 08:14:48.821894 14568 ts_tablet_manager.cc:1909] T c20646678969482994075f88801d101f P d4be1c81b23d41d59b3af62afdf5c4a4: Deleting consensus metadata
I20240503 08:14:48.822361 14019 catalog_manager.cc:4830] TS d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707): tablet c20646678969482994075f88801d101f (table test-table [id=0b86f901f1414dfeac242df4f9411f12]) successfully deleted
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
W20240503 08:14:48.850451 14102 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:48.844790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.6.68.254:43581)
0503 08:14:48.844851 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.844853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.845002 (+ 149us) server_negotiation.cc:407] Connection header received
0503 08:14:48.845083 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.845084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.845092 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.845143 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:48.846058 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.846606 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.847425 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.847552 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.850401 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52588: BlockingRecv error: recv got EOF from 127.0.0.1:52588 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
May 03 08:14:48 dist-test-slave-3g67 krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM
I20240503 08:14:48.857820 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.851470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.6.68.254:43581)
0503 08:14:48.851518 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.851520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.851565 (+ 45us) server_negotiation.cc:407] Connection header received
0503 08:14:48.851654 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.851654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.851665 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.851714 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:48.852251 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.852711 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.853207 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.853392 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.855421 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:48.855425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:48.855426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:48.855432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:48.856870 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.857170 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.857171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.857172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.857193 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:48.857344 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:48.857345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:48.857345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:48.857432 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:48.857521 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.857621 (+ 100us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.857711 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:14:48.859391 14032 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:14:48.859542 14032 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:14:48.860873 14309 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6 (DEFAULT_TABLE table=test-table [id=46444a65979245fab62bbb77f2599037]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:48.860872 14152 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6 (DEFAULT_TABLE table=test-table [id=46444a65979245fab62bbb77f2599037]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:48.860929 14450 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6 (DEFAULT_TABLE table=test-table [id=46444a65979245fab62bbb77f2599037]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:48.861023 14152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.861024 14309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.861054 14450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:48.862354 14546 tablet_bootstrap.cc:492] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d: Bootstrap starting.
I20240503 08:14:48.862417 14545 tablet_bootstrap.cc:492] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4: Bootstrap starting.
I20240503 08:14:48.862731 14546 tablet_bootstrap.cc:654] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.862753 14545 tablet_bootstrap.cc:654] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.863034 14544 tablet_bootstrap.cc:492] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169: Bootstrap starting.
I20240503 08:14:48.863363 14545 tablet_bootstrap.cc:492] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4: No bootstrap required, opened a new log
I20240503 08:14:48.863401 14544 tablet_bootstrap.cc:654] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:48.863407 14545 ts_tablet_manager.cc:1367] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:48.863523 14545 raft_consensus.cc:348] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.863582 14545 raft_consensus.cc:374] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.863602 14545 raft_consensus.cc:729] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4be1c81b23d41d59b3af62afdf5c4a4, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.863685 14545 consensus_queue.cc:260] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.863783 14545 ts_tablet_manager.cc:1398] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.863958 14544 tablet_bootstrap.cc:492] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169: No bootstrap required, opened a new log
I20240503 08:14:48.864007 14544 ts_tablet_manager.cc:1367] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:14:48.864125 14544 raft_consensus.cc:348] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.864166 14544 raft_consensus.cc:374] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.864179 14544 raft_consensus.cc:729] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bda12194ac49d29e0b0db730db7169, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.864215 14544 consensus_queue.cc:260] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.864315 14544 ts_tablet_manager.cc:1398] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.864449 14546 tablet_bootstrap.cc:492] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d: No bootstrap required, opened a new log
I20240503 08:14:48.864500 14546 ts_tablet_manager.cc:1367] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:14:48.864651 14546 raft_consensus.cc:348] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.864710 14546 raft_consensus.cc:374] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:48.864742 14546 raft_consensus.cc:729] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4208988005e8442fa4f538a5ef2b1f5d, State: Initialized, Role: FOLLOWER
I20240503 08:14:48.864797 14546 consensus_queue.cc:260] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.864882 14546 ts_tablet_manager.cc:1398] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:48.986742 14551 raft_consensus.cc:482] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:48.986871 14551 raft_consensus.cc:504] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.987208 14551 leader_election.cc:288] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707), b7bda12194ac49d29e0b0db730db7169 (127.6.68.194:38409)
I20240503 08:14:48.989749 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.987415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:44707 (local address 127.6.68.195:45357)
0503 08:14:48.987531 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.987543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.987610 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.987792 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.987793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.987796 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:48.988054 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.988058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.988647 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.988648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.989528 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.989530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.989589 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.989593 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.989623 (+ 30us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.989662 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20240503 08:14:48.990170 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.987489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:45357 (local address 127.6.68.193:44707)
0503 08:14:48.987536 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.987538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.987652 (+ 114us) server_negotiation.cc:407] Connection header received
0503 08:14:48.987685 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.987686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.987697 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.987784 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:48.988120 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.988579 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.989607 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.990068 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.990076 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.990098 (+ 22us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.990126 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240503 08:14:48.990351 14180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4828825ec94b4eb9df7ffd9bdb4fb6" candidate_uuid: "4208988005e8442fa4f538a5ef2b1f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" is_pre_election: true
I20240503 08:14:48.990429 14180 raft_consensus.cc:2454] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4208988005e8442fa4f538a5ef2b1f5d in term 0.
I20240503 08:14:48.990592 14403 leader_election.cc:302] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4208988005e8442fa4f538a5ef2b1f5d, d4be1c81b23d41d59b3af62afdf5c4a4; no voters:
I20240503 08:14:48.990713 14551 raft_consensus.cc:2781] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:48.990764 14551 raft_consensus.cc:482] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:48.990790 14551 raft_consensus.cc:3037] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.991149 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.987609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38409 (local address 127.6.68.195:36867)
0503 08:14:48.987715 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:48.987751 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:48.987792 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:48.988065 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:48.988066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:48.988069 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:14:48.988636 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.988639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.989457 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.989457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:48.990363 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:48.990365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.990960 (+ 595us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.990963 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:14:48.991028 (+ 65us) client_negotiation.cc:240] Negotiation successful
0503 08:14:48.991069 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":74,"threads_started":1}
I20240503 08:14:48.991168 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:48.987880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:36867 (local address 127.6.68.194:38409)
0503 08:14:48.987943 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:48.987945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:48.987953 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:14:48.987976 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:48.987977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:48.987985 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:48.988037 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:14:48.988712 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.989375 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:48.990468 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:48.990925 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:48.990931 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:48.991060 (+ 129us) server_negotiation.cc:299] Negotiation successful
0503 08:14:48.991108 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240503 08:14:48.991371 14330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4828825ec94b4eb9df7ffd9bdb4fb6" candidate_uuid: "4208988005e8442fa4f538a5ef2b1f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7bda12194ac49d29e0b0db730db7169" is_pre_election: true
I20240503 08:14:48.991457 14330 raft_consensus.cc:2454] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4208988005e8442fa4f538a5ef2b1f5d in term 0.
I20240503 08:14:48.991881 14551 raft_consensus.cc:504] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.992002 14551 leader_election.cc:288] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [CANDIDATE]: Term 1 election: Requested vote from peers d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707), b7bda12194ac49d29e0b0db730db7169 (127.6.68.194:38409)
I20240503 08:14:48.992177 14180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4828825ec94b4eb9df7ffd9bdb4fb6" candidate_uuid: "4208988005e8442fa4f538a5ef2b1f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4"
I20240503 08:14:48.992208 14330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4828825ec94b4eb9df7ffd9bdb4fb6" candidate_uuid: "4208988005e8442fa4f538a5ef2b1f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7bda12194ac49d29e0b0db730db7169"
I20240503 08:14:48.992250 14180 raft_consensus.cc:3037] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.992282 14330 raft_consensus.cc:3037] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:48.992816 14180 raft_consensus.cc:2454] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4208988005e8442fa4f538a5ef2b1f5d in term 1.
I20240503 08:14:48.992975 14403 leader_election.cc:302] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4208988005e8442fa4f538a5ef2b1f5d, d4be1c81b23d41d59b3af62afdf5c4a4; no voters:
I20240503 08:14:48.993018 14330 raft_consensus.cc:2454] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4208988005e8442fa4f538a5ef2b1f5d in term 1.
I20240503 08:14:48.993053 14551 raft_consensus.cc:2781] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:48.993165 14551 raft_consensus.cc:686] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 LEADER]: Becoming Leader. State: Replica: 4208988005e8442fa4f538a5ef2b1f5d, State: Running, Role: LEADER
I20240503 08:14:48.993237 14551 consensus_queue.cc:237] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:48.993927 14031 catalog_manager.cc:5484] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195). New cstate: current_term: 1 leader_uuid: "4208988005e8442fa4f538a5ef2b1f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } health_report { overall_health: HEALTHY } } }
I20240503 08:14:49.012080 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:49.009746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36366 (local address 127.6.68.195:43407)
0503 08:14:49.009787 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:49.009789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:49.009799 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:14:49.009887 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:49.009888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:49.009905 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:49.009974 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:14:49.010271 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:49.010707 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:49.011523 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:49.011716 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:49.011759 (+ 43us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:14:49.011846 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:14:49.011916 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:49.011980 (+ 64us) server_negotiation.cc:299] Negotiation successful
0503 08:14:49.012001 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:14:49.013022 14330 raft_consensus.cc:1261] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Refusing update from remote peer 4208988005e8442fa4f538a5ef2b1f5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:49.013017 14180 raft_consensus.cc:1261] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Refusing update from remote peer 4208988005e8442fa4f538a5ef2b1f5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:49.013275 14614 consensus_queue.cc:1035] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:49.013417 14551 consensus_queue.cc:1035] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:49.017207 14031 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52590:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:14:49.017305 14031 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52590:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:14:49.017925 14031 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744: Sending DeleteTablet for 3 replicas of tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6
I20240503 08:14:49.018251 14152 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:49 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:44162
I20240503 08:14:49.018288 14450 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:49 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:36358
I20240503 08:14:49.018397 14568 tablet_replica.cc:330] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4: stopping tablet replica
I20240503 08:14:49.018452 14568 raft_consensus.cc:2229] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:49.018510 14568 raft_consensus.cc:2258] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:49.018579 14567 tablet_replica.cc:330] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d: stopping tablet replica
I20240503 08:14:49.018630 14567 raft_consensus.cc:2229] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:14:49.018710 14567 raft_consensus.cc:2258] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:49.018985 14309 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:49 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:59090
I20240503 08:14:49.019073 14566 tablet_replica.cc:330] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169: stopping tablet replica
I20240503 08:14:49.019115 14566 raft_consensus.cc:2229] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:49.019160 14566 raft_consensus.cc:2258] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:49.019562 14566 ts_tablet_manager.cc:1875] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:49.021240 14567 ts_tablet_manager.cc:1875] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:49.021436 14566 ts_tablet_manager.cc:1888] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:14:49.021507 14566 log.cc:1196] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/8e4828825ec94b4eb9df7ffd9bdb4fb6
I20240503 08:14:49.021725 14566 ts_tablet_manager.cc:1909] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P b7bda12194ac49d29e0b0db730db7169: Deleting consensus metadata
I20240503 08:14:49.022569 14567 ts_tablet_manager.cc:1888] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:14:49.022629 14567 log.cc:1196] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/8e4828825ec94b4eb9df7ffd9bdb4fb6
I20240503 08:14:49.022841 14567 ts_tablet_manager.cc:1909] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P 4208988005e8442fa4f538a5ef2b1f5d: Deleting consensus metadata
I20240503 08:14:49.023069 14019 catalog_manager.cc:4830] TS b7bda12194ac49d29e0b0db730db7169 (127.6.68.194:38409): tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6 (table test-table [id=46444a65979245fab62bbb77f2599037]) successfully deleted
I20240503 08:14:49.023461 14019 catalog_manager.cc:4830] TS 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407): tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6 (table test-table [id=46444a65979245fab62bbb77f2599037]) successfully deleted
I20240503 08:14:49.023576 14031 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:14:49.023738 14031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:14:49.024250 14568 ts_tablet_manager.cc:1875] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:49.025651 14568 ts_tablet_manager.cc:1888] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:14:49.025714 14568 log.cc:1196] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/8e4828825ec94b4eb9df7ffd9bdb4fb6
I20240503 08:14:49.025925 14568 ts_tablet_manager.cc:1909] T 8e4828825ec94b4eb9df7ffd9bdb4fb6 P d4be1c81b23d41d59b3af62afdf5c4a4: Deleting consensus metadata
I20240503 08:14:49.026429 14019 catalog_manager.cc:4830] TS d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707): tablet 8e4828825ec94b4eb9df7ffd9bdb4fb6 (table test-table [id=46444a65979245fab62bbb77f2599037]) successfully deleted
I20240503 08:14:49.026484 14152 tablet_service.cc:1467] Processing CreateTablet for tablet 6940a1dd30b848da981c9a044a116df3 (DEFAULT_TABLE table=test-table [id=5cd390215f4c4ce191a3f65bb4b473c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:49.026585 14152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6940a1dd30b848da981c9a044a116df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:49.026592 14450 tablet_service.cc:1467] Processing CreateTablet for tablet 6940a1dd30b848da981c9a044a116df3 (DEFAULT_TABLE table=test-table [id=5cd390215f4c4ce191a3f65bb4b473c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:49.026681 14450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6940a1dd30b848da981c9a044a116df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:49.027769 14309 tablet_service.cc:1467] Processing CreateTablet for tablet 6940a1dd30b848da981c9a044a116df3 (DEFAULT_TABLE table=test-table [id=5cd390215f4c4ce191a3f65bb4b473c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:14:49.027866 14309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6940a1dd30b848da981c9a044a116df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:49.029040 14545 tablet_bootstrap.cc:492] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4: Bootstrap starting.
I20240503 08:14:49.029037 14544 tablet_bootstrap.cc:492] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169: Bootstrap starting.
I20240503 08:14:49.029361 14544 tablet_bootstrap.cc:654] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:49.029387 14545 tablet_bootstrap.cc:654] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:49.029953 14545 tablet_bootstrap.cc:492] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4: No bootstrap required, opened a new log
I20240503 08:14:49.029999 14545 ts_tablet_manager.cc:1367] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:49.029999 14546 tablet_bootstrap.cc:492] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d: Bootstrap starting.
I20240503 08:14:49.030103 14545 raft_consensus.cc:348] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:49.030166 14545 raft_consensus.cc:374] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:49.030184 14545 raft_consensus.cc:729] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4be1c81b23d41d59b3af62afdf5c4a4, State: Initialized, Role: FOLLOWER
I20240503 08:14:49.030225 14545 consensus_queue.cc:260] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:49.030306 14545 ts_tablet_manager.cc:1398] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:49.030375 14546 tablet_bootstrap.cc:654] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:49.031070 14546 tablet_bootstrap.cc:492] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d: No bootstrap required, opened a new log
I20240503 08:14:49.031126 14546 ts_tablet_manager.cc:1367] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:49.031278 14546 raft_consensus.cc:348] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:49.031348 14546 raft_consensus.cc:374] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:49.031368 14546 raft_consensus.cc:729] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4208988005e8442fa4f538a5ef2b1f5d, State: Initialized, Role: FOLLOWER
I20240503 08:14:49.031421 14546 consensus_queue.cc:260] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:49.031527 14546 ts_tablet_manager.cc:1398] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:14:49.032282 14544 tablet_bootstrap.cc:492] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169: No bootstrap required, opened a new log
I20240503 08:14:49.032341 14544 ts_tablet_manager.cc:1367] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:14:49.032445 14544 raft_consensus.cc:348] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:49.032502 14544 raft_consensus.cc:374] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:49.032521 14544 raft_consensus.cc:729] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bda12194ac49d29e0b0db730db7169, State: Initialized, Role: FOLLOWER
I20240503 08:14:49.032565 14544 consensus_queue.cc:260] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:49.032660 14544 ts_tablet_manager.cc:1398] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:49.188241 14595 raft_consensus.cc:482] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:14:49.188352 14595 raft_consensus.cc:504] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:49.188499 14595 leader_election.cc:288] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707), 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407)
I20240503 08:14:49.188772 14460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6940a1dd30b848da981c9a044a116df3" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4208988005e8442fa4f538a5ef2b1f5d" is_pre_election: true
I20240503 08:14:49.188766 14180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6940a1dd30b848da981c9a044a116df3" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" is_pre_election: true
I20240503 08:14:49.188858 14180 raft_consensus.cc:2454] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 0.
I20240503 08:14:49.188858 14460 raft_consensus.cc:2454] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 0.
I20240503 08:14:49.189002 14263 leader_election.cc:302] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7bda12194ac49d29e0b0db730db7169, d4be1c81b23d41d59b3af62afdf5c4a4; no voters:
I20240503 08:14:49.189122 14595 raft_consensus.cc:2781] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:14:49.189172 14595 raft_consensus.cc:482] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:14:49.189203 14595 raft_consensus.cc:3037] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:49.189944 14595 raft_consensus.cc:504] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:49.190060 14595 leader_election.cc:288] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 election: Requested vote from peers d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707), 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407)
I20240503 08:14:49.190212 14180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6940a1dd30b848da981c9a044a116df3" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4"
I20240503 08:14:49.190235 14460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6940a1dd30b848da981c9a044a116df3" candidate_uuid: "b7bda12194ac49d29e0b0db730db7169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4208988005e8442fa4f538a5ef2b1f5d"
I20240503 08:14:49.190384 14180 raft_consensus.cc:3037] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:49.190395 14460 raft_consensus.cc:3037] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:49.190991 14180 raft_consensus.cc:2454] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 1.
I20240503 08:14:49.190996 14460 raft_consensus.cc:2454] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7bda12194ac49d29e0b0db730db7169 in term 1.
I20240503 08:14:49.191159 14263 leader_election.cc:302] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4208988005e8442fa4f538a5ef2b1f5d, b7bda12194ac49d29e0b0db730db7169; no voters:
I20240503 08:14:49.191282 14595 raft_consensus.cc:2781] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:49.191344 14595 raft_consensus.cc:686] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 1 LEADER]: Becoming Leader. State: Replica: b7bda12194ac49d29e0b0db730db7169, State: Running, Role: LEADER
I20240503 08:14:49.191402 14595 consensus_queue.cc:237] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } }
I20240503 08:14:49.192027 14033 catalog_manager.cc:5484] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7bda12194ac49d29e0b0db730db7169 (127.6.68.194). New cstate: current_term: 1 leader_uuid: "b7bda12194ac49d29e0b0db730db7169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7bda12194ac49d29e0b0db730db7169" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 } health_report { overall_health: UNKNOWN } } }
I20240503 08:14:49.220052 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:49.218297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.6.68.194:38409)
0503 08:14:49.218362 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:49.218364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:49.218375 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:14:49.218397 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:49.218397 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:49.218406 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:49.218466 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:14:49.218700 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:49.219158 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:49.219678 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:49.219808 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:49.219824 (+ 16us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:14:49.219845 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:14:49.219888 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:49.219946 (+ 58us) server_negotiation.cc:299] Negotiation successful
0503 08:14:49.219962 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240503 08:14:49.222177 14460 raft_consensus.cc:1261] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Refusing update from remote peer b7bda12194ac49d29e0b0db730db7169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:49.222308 14180 raft_consensus.cc:1261] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Refusing update from remote peer b7bda12194ac49d29e0b0db730db7169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:14:49.222441 14550 consensus_queue.cc:1035] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4208988005e8442fa4f538a5ef2b1f5d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:49.222548 14573 consensus_queue.cc:1035] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4be1c81b23d41d59b3af62afdf5c4a4" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 44707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:14:49.231921 14033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52590:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:14:49.232018 14033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52590:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:14:49.232580 14033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 797d178d01474365bcbd22e5304f8744: Sending DeleteTablet for 3 replicas of tablet 6940a1dd30b848da981c9a044a116df3
I20240503 08:14:49.232856 14152 tablet_service.cc:1514] Processing DeleteTablet for tablet 6940a1dd30b848da981c9a044a116df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:49 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:44162
I20240503 08:14:49.232872 14309 tablet_service.cc:1514] Processing DeleteTablet for tablet 6940a1dd30b848da981c9a044a116df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:49 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:59090
I20240503 08:14:49.232955 14568 tablet_replica.cc:330] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4: stopping tablet replica
I20240503 08:14:49.232977 14566 tablet_replica.cc:330] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169: stopping tablet replica
I20240503 08:14:49.233007 14568 raft_consensus.cc:2229] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:49.233026 14566 raft_consensus.cc:2229] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:14:49.233042 14568 raft_consensus.cc:2258] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:49.233084 14566 raft_consensus.cc:2258] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:49.233129 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 14105
I20240503 08:14:49.233414 14568 ts_tablet_manager.cc:1875] T 6940a1dd30b848da981c9a044a116df3 P d4be1c81b23d41d59b3af62afdf5c4a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:49.233668 14450 tablet_service.cc:1514] Processing DeleteTablet for tablet 6940a1dd30b848da981c9a044a116df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:14:49 UTC) from {username='oryx', principal='oryx/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:36358
I20240503 08:14:49.233786 14567 tablet_replica.cc:330] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d: stopping tablet replica
I20240503 08:14:49.233852 14567 raft_consensus.cc:2229] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:14:49.233892 14567 raft_consensus.cc:2258] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:14:49.234165 14567 ts_tablet_manager.cc:1875] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:49.234359 14566 ts_tablet_manager.cc:1875] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:14:49.235901 14567 ts_tablet_manager.cc:1888] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:14:49.235908 14566 ts_tablet_manager.cc:1888] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240503 08:14:49.235957 14566 log.cc:1196] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/6940a1dd30b848da981c9a044a116df3
I20240503 08:14:49.235963 14567 log.cc:1196] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/6940a1dd30b848da981c9a044a116df3
I20240503 08:14:49.236173 14566 ts_tablet_manager.cc:1909] T 6940a1dd30b848da981c9a044a116df3 P b7bda12194ac49d29e0b0db730db7169: Deleting consensus metadata
I20240503 08:14:49.236174 14567 ts_tablet_manager.cc:1909] T 6940a1dd30b848da981c9a044a116df3 P 4208988005e8442fa4f538a5ef2b1f5d: Deleting consensus metadata
I20240503 08:14:49.236457 14019 catalog_manager.cc:4830] TS 4208988005e8442fa4f538a5ef2b1f5d (127.6.68.195:43407): tablet 6940a1dd30b848da981c9a044a116df3 (table test-table [id=5cd390215f4c4ce191a3f65bb4b473c1]) successfully deleted
I20240503 08:14:49.236624 14019 catalog_manager.cc:4830] TS b7bda12194ac49d29e0b0db730db7169 (127.6.68.194:38409): tablet 6940a1dd30b848da981c9a044a116df3 (table test-table [id=5cd390215f4c4ce191a3f65bb4b473c1]) successfully deleted
W20240503 08:14:49.240159 14019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:44707 (error 108)
I20240503 08:14:49.240442 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 14245
I20240503 08:14:49.241091 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:49.240460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:44707 (local address 127.0.0.1:44178)
0503 08:14:49.241009 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:49.241048 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":36,"threads_started":1}
W20240503 08:14:49.241178 14019 catalog_manager.cc:4573] TS d4be1c81b23d41d59b3af62afdf5c4a4 (127.6.68.193:44707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6940a1dd30b848da981c9a044a116df3: Network error: Client connection negotiation failed: client connection to 127.6.68.193:44707: connect: Connection refused (error 111)
I20240503 08:14:49.247574 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 14385
I20240503 08:14:49.253362 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 14001
2024-05-03T08:14:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2634 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:14:49 dist-test-slave-3g67 krb5kdc[14630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:14:49 dist-test-slave-3g67 krb5kdc[14630](info): set up 2 sockets
May 03 08:14:49 dist-test-slave-3g67 krb5kdc[14630](info): commencing operation
krb5kdc: starting...
W20240503 08:14:50.204301 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:14:50 dist-test-slave-3g67 krb5kdc[14630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:14:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:14:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:14:50.324544 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:39061
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:38525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:39061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:50.404731 14643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:50.404914 14643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:50.404944 14643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:50.406214 14643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:50.406270 14643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:50.406288 14643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:50.406307 14643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:50.406325 14643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:14:50.407971 14643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:38525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:39061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:39061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:50.408187 14643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:50.408378 14643 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:50.413095 14651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:50.413571 14648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:50.414254 14649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:50.415129 14643 server_base.cc:1034] running on GCE node
I20240503 08:14:50.415385 14643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:50.415685 14643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:50.416857 14643 hybrid_clock.cc:648] HybridClock initialized: now 1714724090416831 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2024-05-03T08:14:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1178 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:14:50 dist-test-slave-3g67 krb5kdc[14656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:14:50 dist-test-slave-3g67 krb5kdc[14656](info): set up 2 sockets
May 03 08:14:50 dist-test-slave-3g67 krb5kdc[14656](info): commencing operation
krb5kdc: starting...
W20240503 08:14:51.384189 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:14:51 dist-test-slave-3g67 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:14:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:14:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:14:51.503523 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:43449
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:43449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:51.581447 14669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:51.581665 14669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:51.581698 14669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:51.582994 14669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:51.583043 14669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:51.583066 14669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:51.583084 14669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:51.583104 14669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:14:51.584738 14669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:43449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:51.584976 14669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:51.585158 14669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:51.589862 14675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:51.590698 14674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:51.591202 14677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:51.592190 14669 server_base.cc:1034] running on GCE node
I20240503 08:14:51.592481 14669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:51.592723 14669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:51.593900 14669 hybrid_clock.cc:648] HybridClock initialized: now 1714724091593872 us; error 50 us; skew 500 ppm
May 03 08:14:51 dist-test-slave-3g67 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724091, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:51.596410 14669 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:14:51.596786 14669 webserver.cc:460] Webserver started at http://127.6.68.254:44603/ using document root <none> and password file <none>
I20240503 08:14:51.596966 14669 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:51.597007 14669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:51.597100 14669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:51.598079 14669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "5ab7d88a05bd4258a1c494ca0c24340c"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "96a9440627f37693e83111eee603c40e"
server_key_iv: "225bdfd01252052bdc1845c4e7cfc61c"
server_key_version: "encryptionkey@0"
I20240503 08:14:51.598378 14669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "5ab7d88a05bd4258a1c494ca0c24340c"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "96a9440627f37693e83111eee603c40e"
server_key_iv: "225bdfd01252052bdc1845c4e7cfc61c"
server_key_version: "encryptionkey@0"
I20240503 08:14:51.599678 14669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:14:51.600476 14684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.600703 14669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:14:51.600785 14669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "5ab7d88a05bd4258a1c494ca0c24340c"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "96a9440627f37693e83111eee603c40e"
server_key_iv: "225bdfd01252052bdc1845c4e7cfc61c"
server_key_version: "encryptionkey@0"
I20240503 08:14:51.600855 14669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:51.616657 14669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:51.618685 14669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:51.618796 14669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:51.622664 14669 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:43449
I20240503 08:14:51.622712 14736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:43449 every 8 connection(s)
I20240503 08:14:51.623023 14669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:14:51.623608 14737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:14:51.626020 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c: Bootstrap starting.
I20240503 08:14:51.626564 14737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c: Neither blocks nor log segments found. Creating new log.
I20240503 08:14:51.626824 14737 log.cc:824] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:51.627468 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c: No bootstrap required, opened a new log
I20240503 08:14:51.628147 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 14669
I20240503 08:14:51.628257 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:14:51.628444 6419 external_mini_cluster.cc:1428] Setting key bc836e2c0dd95cb9c21b3bc4cc29ee24
I20240503 08:14:51.629261 14737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } }
I20240503 08:14:51.629419 14737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:14:51.629451 14737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab7d88a05bd4258a1c494ca0c24340c, State: Initialized, Role: FOLLOWER
I20240503 08:14:51.629575 14737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } }
I20240503 08:14:51.629654 14737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:14:51.629688 14737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:14:51.629735 14737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:14:51.630270 14737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } }
I20240503 08:14:51.630375 14737 leader_election.cc:302] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ab7d88a05bd4258a1c494ca0c24340c; no voters:
I20240503 08:14:51.630554 14737 leader_election.cc:288] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:14:51.630607 14742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:14:51.630759 14742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 1 LEADER]: Becoming Leader. State: Replica: 5ab7d88a05bd4258a1c494ca0c24340c, State: Running, Role: LEADER
I20240503 08:14:51.630810 14737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:51.630859 14742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } }
I20240503 08:14:51.631152 14742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ab7d88a05bd4258a1c494ca0c24340c. Latest consensus state: current_term: 1 leader_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } } }
I20240503 08:14:51.631254 14742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:51.631166 14744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } } }
I20240503 08:14:51.631407 14744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [sys.catalog]: This master's current role is: LEADER
W20240503 08:14:51.632649 14757 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:14:51.633293 14757 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:14:51.633368 14748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:14:51.633565 14748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
May 03 08:14:51 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:14:51.636080 14748 catalog_manager.cc:1347] Generated new cluster ID: b544237277eb4b3fa81326ed230a9fd5
I20240503 08:14:51.636138 14748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:14:51.639498 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:51.628958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.6.68.254:43449)
0503 08:14:51.629162 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:51.629174 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:51.629201 (+ 27us) server_negotiation.cc:407] Connection header received
0503 08:14:51.629302 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:51.629308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:51.629357 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:51.629510 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:51.630318 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:51.632001 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:51.632853 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:51.633070 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:51.635606 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:51.635614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:51.635614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:51.635669 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:51.637769 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:51.638112 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:51.638114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:51.638115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:51.638167 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:51.638301 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:51.638302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:51.638302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:51.638509 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:51.638615 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:51.639095 (+ 480us) server_negotiation.cc:299] Negotiation successful
0503 08:14:51.639211 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20240503 08:14:51.651398 14748 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:14:51.652176 14748 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:14:51.659044 14748 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c: Generated new TSK 0
I20240503 08:14:51.659230 14748 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:14:51.688465 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43449
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:51.768007 14763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:51.768177 14763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:51.768199 14763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:51.769604 14763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:51.769651 14763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:51.769676 14763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:14:51.771279 14763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:51.771476 14763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:51.771713 14763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:51.776361 14768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:51.776865 14769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:51.777449 14771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:51.778383 14763 server_base.cc:1034] running on GCE node
I20240503 08:14:51.778573 14763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:51.778841 14763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:51.780001 14763 hybrid_clock.cc:648] HybridClock initialized: now 1714724091779975 us; error 42 us; skew 500 ppm
May 03 08:14:51 dist-test-slave-3g67 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724091, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:51.782876 14763 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:14:51.783324 14763 webserver.cc:460] Webserver started at http://127.6.68.193:46419/ using document root <none> and password file <none>
I20240503 08:14:51.783531 14763 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:51.783576 14763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:51.783711 14763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:51.784934 14763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "7a79efeb047241cd8091e27a11957499"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "57afd9446703cb7863b0553a5a0d179b"
server_key_iv: "2954255a9d79305cddec878d89ec8eec"
server_key_version: "encryptionkey@0"
I20240503 08:14:51.785260 14763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "7a79efeb047241cd8091e27a11957499"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "57afd9446703cb7863b0553a5a0d179b"
server_key_iv: "2954255a9d79305cddec878d89ec8eec"
server_key_version: "encryptionkey@0"
I20240503 08:14:51.786806 14763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:14:51.787541 14778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.787736 14763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:14:51.787817 14763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "7a79efeb047241cd8091e27a11957499"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "57afd9446703cb7863b0553a5a0d179b"
server_key_iv: "2954255a9d79305cddec878d89ec8eec"
server_key_version: "encryptionkey@0"
I20240503 08:14:51.787894 14763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:51.800674 14763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:51.802433 14763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:51.802537 14763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:51.802742 14763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:51.803038 14763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:51.803072 14763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.803094 14763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:51.803114 14763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.808594 14763 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:36163
I20240503 08:14:51.808665 14891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:36163 every 8 connection(s)
I20240503 08:14:51.808946 14763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:14:51.813278 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 14763
I20240503 08:14:51.813374 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:14:51.813546 6419 external_mini_cluster.cc:1428] Setting key 7d85f36e4d29e152499a7f1070273db1
May 03 08:14:51 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724091, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:14:51.817473 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:51.809685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:46423 (local address 127.6.68.254:43449)
0503 08:14:51.809762 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:51.809764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:51.809909 (+ 145us) server_negotiation.cc:407] Connection header received
0503 08:14:51.810355 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:51.810356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:51.810369 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:51.810452 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:51.811165 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:51.811673 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:51.812186 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:51.812300 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:51.814763 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:51.814767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:51.814767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:51.814774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:51.816253 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:51.816645 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:51.816646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:51.816646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:51.816669 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:51.816858 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:51.816864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:51.816864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:51.816958 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:51.817020 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:51.817278 (+ 258us) server_negotiation.cc:299] Negotiation successful
0503 08:14:51.817369 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240503 08:14:51.817595 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:51.809639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:46423)
0503 08:14:51.809802 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:51.809827 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:51.810266 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:51.810511 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:51.810514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:51.810801 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:51.811088 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:51.811091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:51.811744 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:51.811745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:51.812114 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:51.812115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:51.812450 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:51.813043 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:51.813059 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:51.814647 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:51.816348 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:51.816355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:51.816366 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:51.816564 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:51.816736 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:51.816737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:51.816739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:51.816795 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:51.817023 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:51.817029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:51.817147 (+ 118us) client_negotiation.cc:769] Sending connection context
0503 08:14:51.817248 (+ 101us) client_negotiation.cc:240] Negotiation successful
0503 08:14:51.817394 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20240503 08:14:51.818148 14892 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43449
I20240503 08:14:51.818267 14892 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:51.818536 14892 heartbeater.cc:511] Master 127.6.68.254:43449 requested a full tablet report, sending...
I20240503 08:14:51.819077 14701 ts_manager.cc:194] Registered new tserver with Master: 7a79efeb047241cd8091e27a11957499 (127.6.68.193:36163)
I20240503 08:14:51.819789 14701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:46423
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:14:51.839401 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43449
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:51.917757 14897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:51.917944 14897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:51.917966 14897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:51.919185 14897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:51.919221 14897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:51.919245 14897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:14:51.920846 14897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:51.921088 14897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:51.921241 14897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:51.925761 14902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:51.926167 14905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:51.926940 14903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:51.927770 14897 server_base.cc:1034] running on GCE node
I20240503 08:14:51.927953 14897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:51.928161 14897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:51.929293 14897 hybrid_clock.cc:648] HybridClock initialized: now 1714724091929274 us; error 35 us; skew 500 ppm
May 03 08:14:51 dist-test-slave-3g67 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724091, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:51.931897 14897 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:14:51.932302 14897 webserver.cc:460] Webserver started at http://127.6.68.194:38319/ using document root <none> and password file <none>
I20240503 08:14:51.932474 14897 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:51.932516 14897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:51.932655 14897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:51.933665 14897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "45a45a16431949af96dedf912f42ccbd"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "81104499678f5405dcae0567b4256b08"
server_key_iv: "80d5c56dca8e212c391ffe2c713a7837"
server_key_version: "encryptionkey@0"
I20240503 08:14:51.933976 14897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "45a45a16431949af96dedf912f42ccbd"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "81104499678f5405dcae0567b4256b08"
server_key_iv: "80d5c56dca8e212c391ffe2c713a7837"
server_key_version: "encryptionkey@0"
I20240503 08:14:51.935112 14897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:14:51.935810 14912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.935995 14897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:51.936071 14897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "45a45a16431949af96dedf912f42ccbd"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "81104499678f5405dcae0567b4256b08"
server_key_iv: "80d5c56dca8e212c391ffe2c713a7837"
server_key_version: "encryptionkey@0"
I20240503 08:14:51.936136 14897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:51.954406 14897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:51.956256 14897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:51.956373 14897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:51.956588 14897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:51.956876 14897 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:51.956915 14897 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.956946 14897 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:51.956966 14897 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:51.962347 14897 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:43483
I20240503 08:14:51.962412 15025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:43483 every 8 connection(s)
I20240503 08:14:51.962682 14897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:14:51.963773 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 14897
I20240503 08:14:51.963878 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:14:51.964066 6419 external_mini_cluster.cc:1428] Setting key ab3a6eb34da57e2ff6842f4d9e0f4122
May 03 08:14:51 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724091, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
I20240503 08:14:51.971525 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:51.963299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:49711 (local address 127.6.68.254:43449)
0503 08:14:51.963381 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:51.963383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:51.963579 (+ 196us) server_negotiation.cc:407] Connection header received
0503 08:14:51.964262 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:51.964263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:51.964277 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:51.964334 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:51.965284 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:51.965755 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:51.966423 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:51.966617 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:51.968898 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:51.968901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:51.968902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:51.968907 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:51.970376 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:51.970736 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:51.970737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:51.970737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:51.970760 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:51.970942 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:51.970943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:51.970944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:51.971035 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:51.971097 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:51.971347 (+ 250us) server_negotiation.cc:299] Negotiation successful
0503 08:14:51.971434 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20240503 08:14:51.971648 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:51.963250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:49711)
0503 08:14:51.963430 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:51.963467 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:51.964125 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:51.964391 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:51.964396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:51.964770 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:51.965190 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:51.965195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:51.965851 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:51.965852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:51.966335 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:51.966337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:51.966756 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:51.967367 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:51.967380 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:51.968796 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:51.970462 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:51.970468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:51.970471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:51.970663 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:51.970832 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:51.970833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:51.970833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:51.970885 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:51.971102 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:51.971107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:51.971217 (+ 110us) client_negotiation.cc:769] Sending connection context
0503 08:14:51.971311 (+ 94us) client_negotiation.cc:240] Negotiation successful
0503 08:14:51.971444 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
I20240503 08:14:51.972148 15026 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43449
I20240503 08:14:51.972236 15026 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:51.972443 15026 heartbeater.cc:511] Master 127.6.68.254:43449 requested a full tablet report, sending...
I20240503 08:14:51.972823 14701 ts_manager.cc:194] Registered new tserver with Master: 45a45a16431949af96dedf912f42ccbd (127.6.68.194:43483)
I20240503 08:14:51.973188 14701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:49711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:14:51.991493 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43449
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:52.072296 15031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:52.072492 15031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:52.072526 15031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:52.073807 15031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:52.073853 15031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:52.073892 15031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:14:52.075439 15031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:52.075685 15031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:52.075877 15031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:52.080564 15036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.081262 15039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.081881 15037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:52.082883 15031 server_base.cc:1034] running on GCE node
I20240503 08:14:52.083151 15031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:52.083377 15031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:52.084512 15031 hybrid_clock.cc:648] HybridClock initialized: now 1714724092084494 us; error 35 us; skew 500 ppm
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724092, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:52.086943 15031 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:14:52.087344 15031 webserver.cc:460] Webserver started at http://127.6.68.195:41773/ using document root <none> and password file <none>
I20240503 08:14:52.087533 15031 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:52.087581 15031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:52.087716 15031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:14:52.088707 15031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "f6cba9a07d3c47579266d9fec6c52868"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-3g67"
server_key: "3928100c5ebac6d811480ddb2e566370"
server_key_iv: "13c3775f98b18b074d513cf6b23ea093"
server_key_version: "encryptionkey@0"
I20240503 08:14:52.089021 15031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "f6cba9a07d3c47579266d9fec6c52868"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-3g67"
server_key: "3928100c5ebac6d811480ddb2e566370"
server_key_iv: "13c3775f98b18b074d513cf6b23ea093"
server_key_version: "encryptionkey@0"
I20240503 08:14:52.090284 15031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:14:52.090992 15046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.091152 15031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:14:52.091235 15031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "f6cba9a07d3c47579266d9fec6c52868"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-3g67"
server_key: "3928100c5ebac6d811480ddb2e566370"
server_key_iv: "13c3775f98b18b074d513cf6b23ea093"
server_key_version: "encryptionkey@0"
I20240503 08:14:52.091301 15031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:52.103163 15031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:52.105254 15031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:52.105360 15031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:52.105517 15031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:52.105751 15031 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:52.105780 15031 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.105803 15031 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:52.105814 15031 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.111078 15031 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:40039
I20240503 08:14:52.111155 15159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:40039 every 8 connection(s)
I20240503 08:14:52.111415 15031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724092, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:14:52.116683 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 15031
I20240503 08:14:52.116789 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:14:52.117000 6419 external_mini_cluster.cc:1428] Setting key 13023a267490ecf23b6227f1047c495a
I20240503 08:14:52.119493 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.112147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:56309 (local address 127.6.68.254:43449)
0503 08:14:52.112220 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:52.112222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:52.112375 (+ 153us) server_negotiation.cc:407] Connection header received
0503 08:14:52.112815 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:52.112816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:52.112825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:52.112886 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:52.113601 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.114085 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.114586 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.114684 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.116809 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:52.116813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:52.116813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:52.116819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:52.118370 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:52.118727 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:52.118728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:52.118729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:52.118748 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:52.118927 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:52.118928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:52.118928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:52.119017 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:52.119075 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:52.119318 (+ 243us) server_negotiation.cc:299] Negotiation successful
0503 08:14:52.119403 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240503 08:14:52.119634 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.112100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:56309)
0503 08:14:52.112268 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:52.112294 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:52.112727 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:52.112943 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:52.112947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:52.113226 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:52.113511 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.113515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.114153 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.114154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:52.114525 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.114526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.114843 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.115321 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:52.115333 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:52.116702 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:14:52.118456 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:52.118462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:52.118465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:52.118656 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:52.118809 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:14:52.118809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:14:52.118810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:14:52.118863 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:14:52.119079 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:14:52.119084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:14:52.119194 (+ 110us) client_negotiation.cc:769] Sending connection context
0503 08:14:52.119288 (+ 94us) client_negotiation.cc:240] Negotiation successful
0503 08:14:52.119428 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20240503 08:14:52.120126 15160 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43449
I20240503 08:14:52.120208 15160 heartbeater.cc:464] Registering TS with master...
I20240503 08:14:52.120427 15160 heartbeater.cc:511] Master 127.6.68.254:43449 requested a full tablet report, sending...
I20240503 08:14:52.120798 14701 ts_manager.cc:194] Registered new tserver with Master: f6cba9a07d3c47579266d9fec6c52868 (127.6.68.195:40039)
I20240503 08:14:52.121246 14701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:56309
I20240503 08:14:52.128741 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab.
Entry for principal oryx/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab.
WARNING: no policy specified for oryx/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab.
Entry for principal oryx/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab.
WARNING: no policy specified for oryx/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab.
Entry for principal oryx/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab.
I20240503 08:14:52.203198 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 14763
I20240503 08:14:52.207764 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 14897
I20240503 08:14:52.212255 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 15031
I20240503 08:14:52.216569 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 14669
I20240503 08:14:52.221782 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:43449
--webserver_interface=127.6.68.254
--webserver_port=44603
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:43449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:14:52.300551 15169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:52.300737 15169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:52.300757 15169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:52.302119 15169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:14:52.302167 15169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:52.302181 15169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:52.302193 15169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:14:52.302204 15169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:14:52.303854 15169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:43449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:52.304037 15169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:52.304234 15169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:52.308889 15177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.309396 15175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.310124 15174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:52.310905 15169 server_base.cc:1034] running on GCE node
I20240503 08:14:52.311113 15169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:52.311342 15169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:52.312471 15169 hybrid_clock.cc:648] HybridClock initialized: now 1714724092312452 us; error 40 us; skew 500 ppm
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724092, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:52.314918 15169 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:14:52.315321 15169 webserver.cc:460] Webserver started at http://127.6.68.254:44603/ using document root <none> and password file <none>
I20240503 08:14:52.315492 15169 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:52.315546 15169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:52.316810 15169 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:52.317456 15184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.317668 15169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:52.317741 15169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "5ab7d88a05bd4258a1c494ca0c24340c"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "96a9440627f37693e83111eee603c40e"
server_key_iv: "225bdfd01252052bdc1845c4e7cfc61c"
server_key_version: "encryptionkey@0"
I20240503 08:14:52.318038 15169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:52.335148 15169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:52.337294 15169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:52.337412 15169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:52.341172 15169 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:43449
I20240503 08:14:52.341223 15236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:43449 every 8 connection(s)
I20240503 08:14:52.341534 15169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:14:52.343016 15237 sys_catalog.cc:263] Verifying existing consensus state
I20240503 08:14:52.343556 15237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c: Bootstrap starting.
I20240503 08:14:52.346320 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 15169
I20240503 08:14:52.346596 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:36163
--local_ip_for_outbound_sockets=127.6.68.193
--tserver_master_addrs=127.6.68.254:43449
--webserver_port=46419
--webserver_interface=127.6.68.193
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.68.193
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:14:52.352684 15237 log.cc:824] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:14:52.354288 15237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:14:52.354555 15237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c: Bootstrap complete.
I20240503 08:14:52.356707 15237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } }
I20240503 08:14:52.356912 15237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab7d88a05bd4258a1c494ca0c24340c, State: Initialized, Role: FOLLOWER
I20240503 08:14:52.357049 15237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } }
I20240503 08:14:52.357122 15237 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:14:52.357159 15237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:14:52.357206 15237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:14:52.358110 15237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } }
I20240503 08:14:52.358206 15237 leader_election.cc:302] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ab7d88a05bd4258a1c494ca0c24340c; no voters:
I20240503 08:14:52.358377 15237 leader_election.cc:288] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 08:14:52.358444 15241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:14:52.358565 15241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [term 2 LEADER]: Becoming Leader. State: Replica: 5ab7d88a05bd4258a1c494ca0c24340c, State: Running, Role: LEADER
I20240503 08:14:52.358616 15237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:14:52.358645 15241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } }
I20240503 08:14:52.358881 15241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ab7d88a05bd4258a1c494ca0c24340c. Latest consensus state: current_term: 2 leader_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } } }
I20240503 08:14:52.358976 15241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:52.358891 15242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7d88a05bd4258a1c494ca0c24340c" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43449 } } }
I20240503 08:14:52.359158 15242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c [sys.catalog]: This master's current role is: LEADER
I20240503 08:14:52.359464 15245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:14:52.359941 15245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:14:52.360074 15245 catalog_manager.cc:1259] Loaded cluster ID: b544237277eb4b3fa81326ed230a9fd5
I20240503 08:14:52.360109 15245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:14:52.361085 15245 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:14:52.361260 15245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ab7d88a05bd4258a1c494ca0c24340c: Loaded TSK: 0
I20240503 08:14:52.361446 15245 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240503 08:14:52.441892 15239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:52.442086 15239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:52.442124 15239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:52.443431 15239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:52.443476 15239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:52.443517 15239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:14:52.445142 15239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.193.keytab
--principal=oryx/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:36163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=46419
--tserver_master_addrs=127.6.68.254:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:52.445389 15239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:52.445586 15239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:52.450937 15264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.451498 15263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.452167 15266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:52.453027 15239 server_base.cc:1034] running on GCE node
I20240503 08:14:52.453239 15239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:52.453505 15239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:52.454689 15239 hybrid_clock.cc:648] HybridClock initialized: now 1714724092454677 us; error 30 us; skew 500 ppm
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724092, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:52.457895 15239 init.cc:377] Logged in from keytab as oryx/127.6.68.193@KRBTEST.COM (short username oryx)
I20240503 08:14:52.458447 15239 webserver.cc:460] Webserver started at http://127.6.68.193:46419/ using document root <none> and password file <none>
I20240503 08:14:52.458683 15239 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:52.458755 15239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:52.460448 15239 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:14:52.461078 15273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.461293 15239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:52.461362 15239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "7a79efeb047241cd8091e27a11957499"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "57afd9446703cb7863b0553a5a0d179b"
server_key_iv: "2954255a9d79305cddec878d89ec8eec"
server_key_version: "encryptionkey@0"
I20240503 08:14:52.461659 15239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:52.482167 15239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:52.484501 15239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:52.484604 15239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:52.484840 15239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:52.485132 15239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:52.485172 15239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.485193 15239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:52.485205 15239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.491148 15239 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:36163
I20240503 08:14:52.491225 15386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:36163 every 8 connection(s)
I20240503 08:14:52.491531 15239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:14:52.491950 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 15239
I20240503 08:14:52.492230 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:43483
--local_ip_for_outbound_sockets=127.6.68.194
--tserver_master_addrs=127.6.68.254:43449
--webserver_port=38319
--webserver_interface=127.6.68.194
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.68.194
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:52.501112 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.492300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:47321)
0503 08:14:52.492729 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:52.492760 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:52.493349 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:52.493862 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:52.493868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:52.494207 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:52.494480 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.494484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.496068 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.496070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:52.496752 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.496755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.497230 (+ 475us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.497934 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:52.497958 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:52.500885 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":87,"threads_started":1}
W20240503 08:14:52.501405 15387 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:43449 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.68.254:43449: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
W20240503 08:14:52.502264 15390 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:52.492540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:47321 (local address 127.6.68.254:43449)
0503 08:14:52.492737 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:52.492750 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:52.492845 (+ 95us) server_negotiation.cc:407] Connection header received
0503 08:14:52.493590 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:52.493595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:52.493643 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:52.493782 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:52.494613 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.495953 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.496870 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.497121 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.501074 (+ 3953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:47321: BlockingRecv error: recv got EOF from 127.6.68.193:47321 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:52.507346 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.501668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:44807)
0503 08:14:52.502404 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:52.502416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:52.502467 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:52.502647 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:52.502649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:52.502811 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:52.503039 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.503041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.504052 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.504053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:52.504653 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.504655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.504732 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.505268 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:52.505274 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:52.507283 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":718}
W20240503 08:14:52.507882 15392 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:52.501750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:44807 (local address 127.6.68.254:43449)
0503 08:14:52.501927 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:52.501929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:52.502520 (+ 591us) server_negotiation.cc:407] Connection header received
0503 08:14:52.502545 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:52.502546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:52.502556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:52.502619 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:52.503150 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.503958 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.504761 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.505337 (+ 576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.507802 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:44807: BlockingRecv error: recv got EOF from 127.6.68.193:44807 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:52.512449 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.507757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:34535)
0503 08:14:52.508021 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:52.508031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:52.508079 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:52.508237 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:52.508238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:52.508377 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:52.508543 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.508545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.509472 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.509472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:52.510017 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.510020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.510100 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.510602 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:52.510606 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:52.512385 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251}
W20240503 08:14:52.512563 15390 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:52.507794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:34535 (local address 127.6.68.254:43449)
0503 08:14:52.507993 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:52.507994 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:52.508130 (+ 136us) server_negotiation.cc:407] Connection header received
0503 08:14:52.508150 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:52.508150 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:52.508160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:52.508212 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:52.508645 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.509378 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.510125 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.510704 (+ 579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.512504 (+ 1800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:34535: BlockingRecv error: recv got EOF from 127.6.68.193:34535 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
W20240503 08:14:52.512842 15387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240503 08:14:52.576073 15391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:52.576275 15391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:52.576313 15391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:52.577629 15391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:52.577680 15391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:52.577716 15391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:14:52.579326 15391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.194.keytab
--principal=oryx/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:43483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=38319
--tserver_master_addrs=127.6.68.254:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:52.579552 15391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:52.579782 15391 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:52.584512 15400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.584933 15398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.585740 15397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:52.586515 15391 server_base.cc:1034] running on GCE node
I20240503 08:14:52.586678 15391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:52.586882 15391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:52.588011 15391 hybrid_clock.cc:648] HybridClock initialized: now 1714724092587998 us; error 30 us; skew 500 ppm
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724092, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:52.590432 15391 init.cc:377] Logged in from keytab as oryx/127.6.68.194@KRBTEST.COM (short username oryx)
I20240503 08:14:52.590838 15391 webserver.cc:460] Webserver started at http://127.6.68.194:38319/ using document root <none> and password file <none>
I20240503 08:14:52.591008 15391 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:52.591065 15391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:52.592339 15391 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:14:52.592849 15407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.593008 15391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:14:52.593079 15391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "45a45a16431949af96dedf912f42ccbd"
format_stamp: "Formatted at 2024-05-03 08:14:51 on dist-test-slave-3g67"
server_key: "81104499678f5405dcae0567b4256b08"
server_key_iv: "80d5c56dca8e212c391ffe2c713a7837"
server_key_version: "encryptionkey@0"
I20240503 08:14:52.593384 15391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:52.611037 15391 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:52.612906 15391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:52.613026 15391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:52.613230 15391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:52.613543 15391 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:52.613576 15391 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.613608 15391 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:52.613627 15391 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.619190 15391 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:43483
I20240503 08:14:52.619230 15520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:43483 every 8 connection(s)
I20240503 08:14:52.619539 15391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:52.625809 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.620232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:38113)
0503 08:14:52.620387 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:52.620418 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:52.620888 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:52.621149 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:52.621153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:52.621494 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:52.621789 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.621794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.622441 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.622442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:52.622814 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.622815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.623156 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.623705 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:52.623722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:52.625683 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
W20240503 08:14:52.625840 15390 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:52.620294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:38113 (local address 127.6.68.254:43449)
0503 08:14:52.620351 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:52.620354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:52.620504 (+ 150us) server_negotiation.cc:407] Connection header received
0503 08:14:52.621001 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:52.621002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:52.621011 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:52.621078 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:52.621871 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.622373 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.622883 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.622987 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.625792 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:38113: BlockingRecv error: recv got EOF from 127.6.68.194:38113 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20240503 08:14:52.626037 15521 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:43449 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.68.254:43449: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
I20240503 08:14:52.627645 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 15391
I20240503 08:14:52.627930 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:40039
--local_ip_for_outbound_sockets=127.6.68.195
--tserver_master_addrs=127.6.68.254:43449
--webserver_port=41773
--webserver_interface=127.6.68.195
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.68.195
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:52.630172 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.626266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:32849)
0503 08:14:52.626327 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:52.626336 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:52.626369 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:52.626496 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:52.626497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:52.626606 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:52.626749 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.626750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.627315 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.627315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:52.627709 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.627710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.627777 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.628300 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:52.628305 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:52.630123 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20240503 08:14:52.630357 15390 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:52.626335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:32849 (local address 127.6.68.254:43449)
0503 08:14:52.626387 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:52.626389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:52.626397 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:14:52.626418 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:52.626419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:52.626428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:52.626478 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:52.626807 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.627255 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.627789 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.627966 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.630245 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:32849: BlockingRecv error: recv got EOF from 127.6.68.194:32849 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:52.634392 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.630432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:47891)
0503 08:14:52.630524 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:52.630534 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:52.630571 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:52.630711 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:52.630712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:52.630826 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:52.630993 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.630995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.631588 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.631589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:52.631971 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.631972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.632033 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.632588 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:52.632592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:52.634319 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20240503 08:14:52.634485 15390 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:52.630544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:47891 (local address 127.6.68.254:43449)
0503 08:14:52.630609 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:52.630612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:52.630619 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:14:52.630639 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:52.630640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:52.630649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:52.630695 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:52.631069 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.631512 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.632051 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.632213 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.634424 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:47891: BlockingRecv error: recv got EOF from 127.6.68.194:47891 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20240503 08:14:52.634625 15521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240503 08:14:52.708338 15524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:14:52.708523 15524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:14:52.708559 15524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:14:52.709847 15524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:14:52.709892 15524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:14:52.709931 15524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:14:52.711565 15524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/krb5kdc/oryx_127.6.68.195.keytab
--principal=oryx/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:40039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=41773
--tserver_master_addrs=127.6.68.254:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:14:52.711813 15524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:14:52.712028 15524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:14:52.716751 15530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.717201 15529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:14:52.717877 15532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:14:52.718786 15524 server_base.cc:1034] running on GCE node
I20240503 08:14:52.718943 15524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:14:52.719125 15524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:14:52.720257 15524 hybrid_clock.cc:648] HybridClock initialized: now 1714724092720245 us; error 25 us; skew 500 ppm
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724092, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:14:52.722681 15524 init.cc:377] Logged in from keytab as oryx/127.6.68.195@KRBTEST.COM (short username oryx)
I20240503 08:14:52.723052 15524 webserver.cc:460] Webserver started at http://127.6.68.195:41773/ using document root <none> and password file <none>
I20240503 08:14:52.723219 15524 fs_manager.cc:359] Metadata directory not provided
I20240503 08:14:52.723269 15524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:14:52.724516 15524 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:14:52.725023 15539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.725189 15524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.725255 15524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "f6cba9a07d3c47579266d9fec6c52868"
format_stamp: "Formatted at 2024-05-03 08:14:52 on dist-test-slave-3g67"
server_key: "3928100c5ebac6d811480ddb2e566370"
server_key_iv: "13c3775f98b18b074d513cf6b23ea093"
server_key_version: "encryptionkey@0"
I20240503 08:14:52.725535 15524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:14:52.736138 15524 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:14:52.737982 15524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:14:52.738080 15524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:14:52.738267 15524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:14:52.738574 15524 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:14:52.738608 15524 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.738631 15524 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:14:52.738642 15524 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:14:52.743810 15524 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:40039
I20240503 08:14:52.743871 15652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:40039 every 8 connection(s)
I20240503 08:14:52.744164 15524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:52.750403 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.744869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:56999)
0503 08:14:52.745025 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:52.745054 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:52.745503 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:52.745735 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:52.745738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:52.746033 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:52.746312 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.746316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.746977 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.746978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:52.747345 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.747347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.747750 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.748241 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:52.748254 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:52.750270 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
W20240503 08:14:52.750429 15390 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:52.744900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:56999 (local address 127.6.68.254:43449)
0503 08:14:52.744974 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:52.744977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:52.745134 (+ 157us) server_negotiation.cc:407] Connection header received
0503 08:14:52.745607 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:52.745608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:52.745621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:52.745677 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:52.746423 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.746890 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.747434 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.747653 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.750380 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:56999: BlockingRecv error: recv got EOF from 127.6.68.195:56999 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20240503 08:14:52.750620 15653 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:43449 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.68.254:43449: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
I20240503 08:14:52.752920 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 15524
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:52.755043 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.750827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:58355)
0503 08:14:52.750878 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:52.750888 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:52.750918 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:52.751161 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:52.751162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:52.751309 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:52.751516 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.751518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.752370 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.752371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:52.752735 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.752736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.752793 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.753270 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:52.753274 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:52.754997 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20240503 08:14:52.755162 15390 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:52.750979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:58355 (local address 127.6.68.254:43449)
0503 08:14:52.751031 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:52.751033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:52.751039 (+ 6us) server_negotiation.cc:407] Connection header received
0503 08:14:52.751056 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:52.751057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:52.751066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:52.751125 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:52.751606 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.752297 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.752823 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.752994 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.755102 (+ 2108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:58355: BlockingRecv error: recv got EOF from 127.6.68.195:58355 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:52 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:52.759439 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.755281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:36647)
0503 08:14:52.755432 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:52.755443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:52.755511 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:52.755645 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:52.755645 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:52.755816 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:52.756063 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.756065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.756633 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.756633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:52.757005 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:52.757006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.757077 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.757504 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:52.757507 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:52.759302 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
W20240503 08:14:52.759469 15390 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:52.755418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:36647 (local address 127.6.68.254:43449)
0503 08:14:52.755456 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:52.755458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:52.755519 (+ 61us) server_negotiation.cc:407] Connection header received
0503 08:14:52.755555 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:52.755555 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:52.755562 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:52.755605 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:52.756129 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.756556 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.757095 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.757210 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.759407 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:36647: BlockingRecv error: recv got EOF from 127.6.68.195:36647 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20240503 08:14:52.759716 15653 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240503 08:14:52.760334 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:52.753327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.6.68.254:43449)
0503 08:14:52.753368 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:52.753369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:52.753593 (+ 224us) server_negotiation.cc:407] Connection header received
0503 08:14:52.753661 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:52.753661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:52.753671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:52.753727 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:52.754444 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.755054 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:52.755932 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:52.756049 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:52.757050 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:14:52.757057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:14:52.757057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:14:52.757092 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:14:52.759049 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:52.759441 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:52.759443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:52.759444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:52.759493 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:14:52.759726 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:14:52.759727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:14:52.759728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:14:52.759953 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:14:52.760065 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0503 08:14:52.760178 (+ 113us) server_negotiation.cc:299] Negotiation successful
0503 08:14:52.760271 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
May 03 08:14:53 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:53 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:53.517635 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:53.513336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:59075)
0503 08:14:53.513523 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:53.513537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:53.513600 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:53.513736 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:53.513736 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:53.513908 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:53.514128 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:53.514131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.514763 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:53.514763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:53.515127 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:53.515128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.515180 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:53.515715 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:53.515719 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:53.517556 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
W20240503 08:14:53.517771 15659 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:53.513466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:59075 (local address 127.6.68.254:43449)
0503 08:14:53.513592 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:53.513594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:53.513604 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:14:53.513646 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:53.513647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:53.513656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:53.513717 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:53.514201 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.514696 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:53.515188 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.515295 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:53.517693 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:59075: BlockingRecv error: recv got EOF from 127.6.68.193:59075 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
May 03 08:14:53 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:53 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:53.639221 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:53.635171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:39421)
0503 08:14:53.635346 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:53.635358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:53.635409 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:53.635572 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:53.635573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:53.635749 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:53.635949 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:53.635950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.636564 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:53.636565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:53.636931 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:53.636932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.636983 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:53.637466 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:53.637470 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:53.639134 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
W20240503 08:14:53.639374 15659 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:53.635310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:39421 (local address 127.6.68.254:43449)
0503 08:14:53.635380 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:53.635382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:53.635404 (+ 22us) server_negotiation.cc:407] Connection header received
0503 08:14:53.635478 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:53.635479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:53.635491 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:53.635551 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:53.636014 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.636498 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:53.636994 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.637095 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:53.639233 (+ 2138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:39421: BlockingRecv error: recv got EOF from 127.6.68.194:39421 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
May 03 08:14:53 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:53 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:53.764745 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:53.760143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:57821)
0503 08:14:53.760263 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:53.760276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:53.760364 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:53.760583 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:53.760583 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:53.760807 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:53.761005 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:53.761007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.761697 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:53.761699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:53.762083 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:53.762084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.762167 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:53.762797 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:53.762801 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:53.764688 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
W20240503 08:14:53.764889 15659 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:53.760215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:57821 (local address 127.6.68.254:43449)
0503 08:14:53.760265 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:53.760268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:53.760360 (+ 92us) server_negotiation.cc:407] Connection header received
0503 08:14:53.760489 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:53.760490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:53.760502 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:53.760552 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:53.761097 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.761606 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:53.762173 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:53.762362 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:53.764816 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:57821: BlockingRecv error: recv got EOF from 127.6.68.195:57821 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
May 03 08:14:54 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:54 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:54.522120 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:54.518046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:59549)
0503 08:14:54.518189 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:54.518200 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:54.518252 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:54.518376 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:54.518376 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:54.518535 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:54.518694 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:54.518695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.519299 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:54.519299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:54.519705 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:54.519707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.519761 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:54.520275 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:54.520282 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:54.522060 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
W20240503 08:14:54.522243 15663 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:54.518132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:59549 (local address 127.6.68.254:43449)
0503 08:14:54.518248 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:54.518249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:54.518259 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:14:54.518299 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:54.518299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:54.518309 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:54.518360 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:54.518753 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.519240 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:54.519770 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.519896 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:54.522157 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:59549: BlockingRecv error: recv got EOF from 127.6.68.193:59549 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
May 03 08:14:54 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:54 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:54.643707 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:54.639753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:60941)
0503 08:14:54.639918 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:54.639929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:54.639991 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:54.640112 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:54.640112 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:54.640274 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:54.640436 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:54.640438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.641058 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:54.641058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:54.641419 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:54.641420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.641474 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:54.641988 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:54.641992 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:54.643603 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
W20240503 08:14:54.643826 15663 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:54.639865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:60941 (local address 127.6.68.254:43449)
0503 08:14:54.639930 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:54.639932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:54.639986 (+ 54us) server_negotiation.cc:407] Connection header received
0503 08:14:54.640037 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:54.640038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:54.640048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:54.640100 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:54.640495 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.640998 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:54.641481 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.641592 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:54.643736 (+ 2144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:60941: BlockingRecv error: recv got EOF from 127.6.68.194:60941 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
May 03 08:14:54 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:54 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:54.769953 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:54.765199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:56223)
0503 08:14:54.765351 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:54.765363 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:54.765424 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:54.765561 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:54.765562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:54.765747 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:54.765910 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:54.765913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.766650 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:54.766651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:54.767124 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:54.767125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.767233 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:54.767966 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:54.767971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:54.769863 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
W20240503 08:14:54.770038 15663 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:54.765325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:56223 (local address 127.6.68.254:43449)
0503 08:14:54.765366 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:54.765368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:54.765406 (+ 38us) server_negotiation.cc:407] Connection header received
0503 08:14:54.765487 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:54.765488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:54.765500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:54.765555 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:54.765993 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.766566 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:54.767234 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:54.767437 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:54.769979 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:56223: BlockingRecv error: recv got EOF from 127.6.68.195:56223 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
May 03 08:14:55 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:55 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:55.531479 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:55.522610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:60329)
0503 08:14:55.522768 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:55.522783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:55.522841 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:55.522982 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:55.522982 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:55.523144 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:55.523303 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:55.523305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.523970 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:55.523971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:55.524338 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:55.524339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.524406 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:55.529588 (+ 5182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:55.529593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:55.531419 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
W20240503 08:14:55.531608 15667 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:55.522677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:60329 (local address 127.6.68.254:43449)
0503 08:14:55.522809 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:55.522811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:55.522823 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:14:55.522902 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:55.522902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:55.522913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:55.522972 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:55.523365 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.523903 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:55.524402 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.524513 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:55.531532 (+ 7019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:60329: BlockingRecv error: recv got EOF from 127.6.68.193:60329 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
May 03 08:14:55 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:55 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:55.648463 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:55.644344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:48047)
0503 08:14:55.644509 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:55.644519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:55.644584 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:55.644720 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:55.644720 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:55.644893 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:55.645053 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:55.645054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.645653 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:55.645653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:55.646019 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:55.646019 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.646070 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:55.646580 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:55.646583 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:55.648403 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":70,"threads_started":1}
W20240503 08:14:55.648573 15667 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:55.644409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:48047 (local address 127.6.68.254:43449)
0503 08:14:55.644465 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:55.644466 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:55.644582 (+ 116us) server_negotiation.cc:407] Connection header received
0503 08:14:55.644633 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:55.644634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:55.644645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:55.644705 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:55.645115 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.645594 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:55.646080 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.646183 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:55.648507 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:48047: BlockingRecv error: recv got EOF from 127.6.68.194:48047 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
May 03 08:14:55 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:55 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:55.775195 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:55.770406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:51577)
0503 08:14:55.770529 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:55.770539 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:55.770585 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:55.770778 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:55.770779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:55.770948 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:55.771105 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:55.771107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.771711 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:55.771712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:55.772150 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:55.772152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.772254 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:55.772849 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:55.772854 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:55.775086 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
W20240503 08:14:55.775259 15667 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:55.770560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:51577 (local address 127.6.68.254:43449)
0503 08:14:55.770653 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:55.770655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:55.770665 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:14:55.770687 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:55.770688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:55.770700 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:55.770763 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:55.771169 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.771652 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:55.772286 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:55.772519 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:55.775191 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:51577: BlockingRecv error: recv got EOF from 127.6.68.195:51577 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
May 03 08:14:56 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:56 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:56.536170 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:56.531949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:49019)
0503 08:14:56.532100 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:56.532111 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:56.532169 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:56.532293 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:56.532293 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:56.532452 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:56.532607 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:56.532609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.533255 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:56.533256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:56.533633 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:56.533634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.533696 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:56.534238 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:56.534242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:56.536112 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
W20240503 08:14:56.536321 15671 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:56.532079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:49019 (local address 127.6.68.254:43449)
0503 08:14:56.532185 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:56.532186 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:56.532194 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:14:56.532215 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:56.532215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:56.532226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:56.532284 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:56.532662 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.533149 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:56.533701 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.533825 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:56.536212 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:49019: BlockingRecv error: recv got EOF from 127.6.68.193:49019 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
May 03 08:14:56 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:56 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:56.652757 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:56.648955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:52061)
0503 08:14:56.649095 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:56.649105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:56.649160 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:56.649289 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:56.649289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:56.649453 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:56.649612 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:56.649614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.650204 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:56.650204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:56.650557 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:56.650558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.650607 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:56.651080 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:56.651084 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:56.652698 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
W20240503 08:14:56.652876 15671 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:56.649045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:52061 (local address 127.6.68.254:43449)
0503 08:14:56.649103 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:56.649105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:56.649161 (+ 56us) server_negotiation.cc:407] Connection header received
0503 08:14:56.649221 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:56.649222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:56.649232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:56.649275 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:56.649672 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.650147 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:56.650613 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.650713 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:56.652794 (+ 2081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:52061: BlockingRecv error: recv got EOF from 127.6.68.194:52061 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
May 03 08:14:56 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:56 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:56.779875 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:56.775980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:55565)
0503 08:14:56.776156 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:56.776166 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:56.776226 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:56.776345 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:56.776346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:56.776505 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:56.776661 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:56.776662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.777241 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:56.777241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:56.777590 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:56.777591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.777642 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:56.778114 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:56.778117 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:56.779806 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
W20240503 08:14:56.779979 15671 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:56.776039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:55565 (local address 127.6.68.254:43449)
0503 08:14:56.776095 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:56.776096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:56.776223 (+ 127us) server_negotiation.cc:407] Connection header received
0503 08:14:56.776268 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:56.776268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:56.776279 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:56.776328 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:56.776716 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.777183 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:56.777645 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:56.777742 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:56.779911 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:55565: BlockingRecv error: recv got EOF from 127.6.68.195:55565 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
May 03 08:14:57 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:57 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:57.540855 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:57.536671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:56305)
0503 08:14:57.536794 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:57.536817 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:57.536868 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:57.537031 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:57.537033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:57.537358 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:57.537533 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:57.537536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.538161 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:57.538162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:57.538533 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:57.538535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.538587 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:57.539078 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:57.539082 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:57.540774 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
W20240503 08:14:57.540977 15675 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:57.536742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:56305 (local address 127.6.68.254:43449)
0503 08:14:57.536856 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:57.536858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:57.536869 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:14:57.536927 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:57.536927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:57.536939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:57.537016 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:57.537612 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.538101 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:57.538589 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.538692 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:57.540884 (+ 2192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:56305: BlockingRecv error: recv got EOF from 127.6.68.193:56305 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
May 03 08:14:57 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:57 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:57.657563 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:57.653247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:59951)
0503 08:14:57.653379 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:57.653393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:57.653458 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:57.653609 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:57.653610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:57.653800 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:57.653967 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:57.653969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.654619 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:57.654620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:57.654988 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:57.654989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.655053 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:57.655687 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:57.655691 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:57.657506 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
W20240503 08:14:57.657698 15675 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:57.653362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:59951 (local address 127.6.68.254:43449)
0503 08:14:57.653434 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:57.653436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:57.653448 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:14:57.653529 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:57.653530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:57.653545 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:57.653597 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:57.654053 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.654540 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:57.655062 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.655253 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:57.657632 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:59951: BlockingRecv error: recv got EOF from 127.6.68.194:59951 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
May 03 08:14:57 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:57 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:57.784600 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:57.780544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:41177)
0503 08:14:57.780700 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:57.780710 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:57.780771 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:57.780891 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:57.780892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:57.781072 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:57.781232 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:57.781234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.781829 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:57.781829 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:57.782268 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:57.782270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.782331 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:57.782872 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:57.782875 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:57.784521 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
W20240503 08:14:57.784688 15675 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:57.780602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:41177 (local address 127.6.68.254:43449)
0503 08:14:57.780658 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:57.780660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:57.780766 (+ 106us) server_negotiation.cc:407] Connection header received
0503 08:14:57.780814 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:57.780815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:57.780826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:57.780877 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:57.781292 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.781768 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:57.782346 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:57.782476 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:57.784627 (+ 2151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:41177: BlockingRecv error: recv got EOF from 127.6.68.195:41177 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
May 03 08:14:58 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:58 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:58.545828 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:58.541416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:57679)
0503 08:14:58.541563 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:58.541573 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:58.541623 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:58.541774 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:58.541774 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:58.541935 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:58.542092 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:58.542094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.542893 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:58.542894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:58.543480 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:58.543483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.543548 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:58.544098 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:58.544102 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:58.545769 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
W20240503 08:14:58.545934 15679 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:58.541523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:57679 (local address 127.6.68.254:43449)
0503 08:14:58.541641 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:58.541643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:58.541653 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:14:58.541678 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:58.541679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:58.541689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:58.541762 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:58.542157 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.542759 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:58.543556 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.543725 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:58.545863 (+ 2138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:57679: BlockingRecv error: recv got EOF from 127.6.68.193:57679 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
May 03 08:14:58 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:58 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:58.662372 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:58.658118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:38281)
0503 08:14:58.658231 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:58.658244 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:58.658323 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:58.658520 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:58.658520 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:58.658698 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:58.658853 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:58.658855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.659574 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:58.659575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:58.659994 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:58.659995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.660061 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:58.660612 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:58.660616 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:58.662236 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
W20240503 08:14:58.662393 15679 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:58.658197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:38281 (local address 127.6.68.254:43449)
0503 08:14:58.658243 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:58.658245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:58.658323 (+ 78us) server_negotiation.cc:407] Connection header received
0503 08:14:58.658426 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:58.658427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:58.658440 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:58.658509 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:58.658934 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.659489 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:58.660070 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.660252 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:58.662334 (+ 2082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:38281: BlockingRecv error: recv got EOF from 127.6.68.194:38281 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
May 03 08:14:58 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:58 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:58.789326 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:58.785060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:34901)
0503 08:14:58.785164 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:58.785176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:58.785272 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:58.785409 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:58.785410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:58.785579 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:58.785737 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:58.785739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.786442 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:58.786443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:58.786844 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:58.786846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.786911 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:58.787530 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:58.787534 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:58.789258 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
W20240503 08:14:58.789444 15679 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:58.785142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:34901 (local address 127.6.68.254:43449)
0503 08:14:58.785207 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:58.785209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:58.785219 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:14:58.785332 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:58.785333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:58.785346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:58.785399 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:58.785814 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.786362 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:58.786921 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:58.787107 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:58.789366 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:34901: BlockingRecv error: recv got EOF from 127.6.68.195:34901 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
May 03 08:14:59 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:59 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:59.550683 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:59.546458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:41867)
0503 08:14:59.546601 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:59.546613 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:59.546672 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:59.546854 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:59.546855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:59.547070 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:59.547253 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:59.547255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.547944 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:59.547945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:59.548350 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:59.548352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.548416 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:59.548923 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:59.548927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:59.550622 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
W20240503 08:14:59.550803 15683 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:59.546539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:41867 (local address 127.6.68.254:43449)
0503 08:14:59.546654 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:59.546656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:59.546667 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:14:59.546741 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:59.546742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:59.546757 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:59.546829 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:59.547318 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.547852 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:59.548422 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.548526 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:59.550731 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:41867: BlockingRecv error: recv got EOF from 127.6.68.193:41867 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":48,"threads_started":1}
May 03 08:14:59 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:59 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:59.667109 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:59.662855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:55397)
0503 08:14:59.662986 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:59.662996 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:59.663055 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:59.663174 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:59.663175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:59.663337 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:59.663503 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:59.663505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.664188 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:59.664189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:59.664594 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:59.664596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.664658 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:59.665226 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:59.665231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:59.667048 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
W20240503 08:14:59.667205 15683 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:59.662919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:55397 (local address 127.6.68.254:43449)
0503 08:14:59.662979 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:59.662981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:59.663051 (+ 70us) server_negotiation.cc:407] Connection header received
0503 08:14:59.663100 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:59.663101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:59.663110 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:59.663163 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:59.663563 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.664101 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:59.664664 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.664801 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:59.667140 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:55397: BlockingRecv error: recv got EOF from 127.6.68.194:55397 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
May 03 08:14:59 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:14:59 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:14:59.793709 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:14:59.789816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:40589)
0503 08:14:59.789968 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0503 08:14:59.789982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:14:59.790051 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:14:59.790183 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:14:59.790184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:14:59.790336 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:14:59.790493 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:59.790495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.791110 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:14:59.791110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:14:59.791463 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:14:59.791464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.791517 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:59.792035 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:14:59.792038 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:14:59.793650 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
W20240503 08:14:59.793859 15683 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:14:59.789926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:40589 (local address 127.6.68.254:43449)
0503 08:14:59.789997 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0503 08:14:59.789999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:14:59.790035 (+ 36us) server_negotiation.cc:407] Connection header received
0503 08:14:59.790111 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:14:59.790112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:14:59.790123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:14:59.790169 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:14:59.790559 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.791045 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:14:59.791527 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:14:59.791677 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:14:59.793768 (+ 2091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:40589: BlockingRecv error: recv got EOF from 127.6.68.195:40589 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
May 03 08:15:00 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:00 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:00.560915 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:00.551249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:55473)
0503 08:15:00.551409 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:00.551419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:00.551471 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:00.551672 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:00.551673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:00.551852 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:00.552062 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:00.552066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.552836 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:00.552837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:00.553241 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:00.553243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.553312 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:00.559055 (+ 5743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:00.559059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:00.560839 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
W20240503 08:15:00.561069 15687 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:00.551388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:55473 (local address 127.6.68.254:43449)
0503 08:15:00.551513 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:00.551515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:00.551523 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:15:00.551549 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:00.551550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:00.551560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:00.551652 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:00.552157 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.552759 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:00.553315 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.553518 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:00.561000 (+ 7482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:55473: BlockingRecv error: recv got EOF from 127.6.68.193:55473 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
May 03 08:15:00 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:00 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:00.671893 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:00.667736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:60195)
0503 08:15:00.667870 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:00.667880 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:00.667931 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:00.668078 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:00.668079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:00.668239 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:00.668400 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:00.668402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.669182 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:00.669182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:00.669548 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:00.669549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.669604 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:00.670117 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:00.670122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:00.671832 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
W20240503 08:15:00.671964 15687 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:00.667830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:60195 (local address 127.6.68.254:43449)
0503 08:15:00.667910 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:00.667911 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:00.667922 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:15:00.667992 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:00.667993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:00.668002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:00.668063 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:00.668473 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.669120 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:00.669609 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.669709 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:00.671910 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:60195: BlockingRecv error: recv got EOF from 127.6.68.194:60195 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
May 03 08:15:00 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:00 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:00.798403 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:00.794180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:44345)
0503 08:15:00.794306 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:00.794319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:00.794389 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:00.794537 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:00.794538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:00.794712 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:00.794874 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:00.794875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.795522 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:00.795523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:00.796003 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:00.796005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.796076 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:00.796615 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:00.796619 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:00.798343 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
W20240503 08:15:00.798533 15687 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:00.794250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:44345 (local address 127.6.68.254:43449)
0503 08:15:00.794310 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:00.794313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:00.794377 (+ 64us) server_negotiation.cc:407] Connection header received
0503 08:15:00.794460 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:00.794461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:00.794477 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:00.794526 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:00.794962 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.795441 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:00.796106 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:00.796297 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:00.798451 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:44345: BlockingRecv error: recv got EOF from 127.6.68.195:44345 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
May 03 08:15:01 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:01 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:01.565838 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:01.561520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:59049)
0503 08:15:01.561656 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:01.561667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:01.561718 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:01.561886 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:01.561886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:01.562051 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:01.562211 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:01.562213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.562840 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:01.562841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:01.563270 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:01.563272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.563342 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:01.564034 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:01.564039 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:01.565779 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
W20240503 08:15:01.565964 15691 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:01.561611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:59049 (local address 127.6.68.254:43449)
0503 08:15:01.561750 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:01.561752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:01.561762 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:15:01.561787 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:01.561787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:01.561798 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:01.561869 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:01.562279 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.562776 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:01.563346 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.563512 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:01.565889 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:59049: BlockingRecv error: recv got EOF from 127.6.68.193:59049 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
May 03 08:15:01 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:01 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:01.676854 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:01.672544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:39715)
0503 08:15:01.672733 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:01.672746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:01.672827 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:01.672973 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:01.672974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:01.673205 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:01.673403 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:01.673405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.674058 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:01.674058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:01.674447 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:01.674448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.674516 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:01.675033 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:01.675037 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:01.676749 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
W20240503 08:15:01.676915 15691 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:01.672664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:39715 (local address 127.6.68.254:43449)
0503 08:15:01.672734 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:01.672736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:01.672827 (+ 91us) server_negotiation.cc:407] Connection header received
0503 08:15:01.672888 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:01.672889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:01.672900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:01.672956 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:01.673488 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.673980 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:01.674526 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.674715 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:01.676848 (+ 2133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:39715: BlockingRecv error: recv got EOF from 127.6.68.194:39715 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
May 03 08:15:01 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:01 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:01.803179 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:01.798884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:48301)
0503 08:15:01.799041 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:01.799055 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:01.799125 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:01.799272 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:01.799273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:01.799450 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:01.799602 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:01.799604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.800269 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:01.800271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:01.800631 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:01.800632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.800700 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:01.801265 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:01.801269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:01.803100 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
W20240503 08:15:01.803263 15691 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:01.799002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:48301 (local address 127.6.68.254:43449)
0503 08:15:01.799072 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:01.799075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:01.799105 (+ 30us) server_negotiation.cc:407] Connection header received
0503 08:15:01.799187 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:01.799188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:01.799205 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:01.799252 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:01.799705 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.800190 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:01.800708 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:01.800886 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:01.803203 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:48301: BlockingRecv error: recv got EOF from 127.6.68.195:48301 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
May 03 08:15:02 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:02 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:02.570458 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:02.566335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:60571)
0503 08:15:02.566473 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:02.566484 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:02.566535 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:02.566675 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:02.566676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:02.566840 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:02.567025 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:02.567027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.567848 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:02.567849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:02.568225 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:02.568227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.568287 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:02.568799 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:02.568803 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:02.570399 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
W20240503 08:15:02.570590 15695 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:02.566417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:60571 (local address 127.6.68.254:43449)
0503 08:15:02.566543 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:02.566545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:02.566555 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:15:02.566581 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:02.566582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:02.566591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:02.566667 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:02.567103 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.567784 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:02.568292 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.568399 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:02.570512 (+ 2113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:60571: BlockingRecv error: recv got EOF from 127.6.68.193:60571 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
May 03 08:15:02 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:02 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:02.681309 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:02.677312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:48167)
0503 08:15:02.677445 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:02.677456 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:02.677512 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:02.677644 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:02.677644 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:02.677809 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:02.677974 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:02.677975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.678573 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:02.678574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:02.678944 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:02.678946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.678996 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:02.679489 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:02.679493 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:02.681241 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
W20240503 08:15:02.681412 15695 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:02.677419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:48167 (local address 127.6.68.254:43449)
0503 08:15:02.677487 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:02.677489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:02.677499 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:15:02.677568 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:02.677568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:02.677579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:02.677628 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:02.678032 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.678514 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:02.679004 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.679106 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:02.681339 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:48167: BlockingRecv error: recv got EOF from 127.6.68.194:48167 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
May 03 08:15:02 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:02 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:02.808059 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:02.803781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:46047)
0503 08:15:02.803929 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:02.803939 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:02.804000 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:02.804133 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:02.804134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:02.804312 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:02.804467 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:02.804469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.805100 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:02.805101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:02.805458 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:02.805460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.805524 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:02.806110 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:02.806114 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:02.808002 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
W20240503 08:15:02.808238 15695 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:02.803934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:46047 (local address 127.6.68.254:43449)
0503 08:15:02.804010 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:02.804012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:02.804023 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:15:02.804053 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:02.804054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:02.804071 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:02.804118 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:02.804548 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.805021 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:02.805534 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:02.805711 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:02.808138 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:46047: BlockingRecv error: recv got EOF from 127.6.68.195:46047 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
May 03 08:15:03 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:03 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:03.575578 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:03.571015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:40739)
0503 08:15:03.571235 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:03.571248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:03.571328 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:03.571473 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:03.571473 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:03.571731 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:03.571896 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:03.571899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.572613 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:03.572614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:03.573020 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:03.573021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.573084 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:03.573624 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:03.573628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:03.575487 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
W20240503 08:15:03.575714 15699 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:03.571137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:40739 (local address 127.6.68.254:43449)
0503 08:15:03.571256 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:03.571258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:03.571322 (+ 64us) server_negotiation.cc:407] Connection header received
0503 08:15:03.571379 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:03.571380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:03.571391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:03.571453 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:03.572009 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.572516 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:03.573091 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.573197 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:03.575600 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:40739: BlockingRecv error: recv got EOF from 127.6.68.193:40739 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":53,"threads_started":1}
May 03 08:15:03 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:03 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:03.686100 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:03.681874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:49585)
0503 08:15:03.682054 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:03.682064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:03.682126 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:03.682267 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:03.682267 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:03.682428 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:03.682586 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:03.682588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.683228 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:03.683229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:03.683580 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:03.683581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.683642 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:03.684194 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:03.684198 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:03.686041 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
W20240503 08:15:03.686245 15699 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:03.681997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:49585 (local address 127.6.68.254:43449)
0503 08:15:03.682059 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:03.682061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:03.682123 (+ 62us) server_negotiation.cc:407] Connection header received
0503 08:15:03.682176 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:03.682176 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:03.682188 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:03.682249 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:03.682660 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.683161 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:03.683655 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.683781 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:03.686175 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:49585: BlockingRecv error: recv got EOF from 127.6.68.194:49585 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
May 03 08:15:03 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:03 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:03.813041 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:03.808587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:60103)
0503 08:15:03.808727 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:03.808736 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:03.808810 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:03.808933 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:03.808934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:03.809098 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:03.809272 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:03.809275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.809910 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:03.809912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:03.810607 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:03.810611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.810708 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:03.811271 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:03.811275 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:03.812980 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
W20240503 08:15:03.813143 15699 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:03.808672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:60103 (local address 127.6.68.254:43449)
0503 08:15:03.808733 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:03.808735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:03.808810 (+ 75us) server_negotiation.cc:407] Connection header received
0503 08:15:03.808853 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:03.808854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:03.808866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:03.808923 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:03.809336 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.809822 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:03.810707 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:03.810881 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:03.813078 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:60103: BlockingRecv error: recv got EOF from 127.6.68.195:60103 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
May 03 08:15:04 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:04 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:04.579946 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:04.576057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:55249)
0503 08:15:04.576199 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:04.576209 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:04.576258 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:04.576391 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:04.576392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:04.576556 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:04.576708 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:04.576710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.577356 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:04.577356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:04.577708 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:04.577710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.577763 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:04.578254 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:04.578258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:04.579879 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
W20240503 08:15:04.580068 15703 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:04.576124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:55249 (local address 127.6.68.254:43449)
0503 08:15:04.576231 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:04.576233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:04.576243 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:15:04.576309 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:04.576310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:04.576319 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:04.576377 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:04.576766 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.577280 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:04.577770 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.577870 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:04.579971 (+ 2101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:55249: BlockingRecv error: recv got EOF from 127.6.68.193:55249 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
May 03 08:15:04 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:04 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:04.690838 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:04.686823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:46399)
0503 08:15:04.686978 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:04.686989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:04.687046 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:04.687210 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:04.687210 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:04.687374 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:04.687539 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:04.687540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.688164 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:04.688165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:04.688523 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:04.688524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.688580 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:04.689096 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:04.689101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:04.690766 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
W20240503 08:15:04.690984 15703 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:04.686915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:46399 (local address 127.6.68.254:43449)
0503 08:15:04.687015 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:04.687017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:04.687029 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:15:04.687121 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:04.687122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:04.687134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:04.687196 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:04.687601 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.688103 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:04.688584 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.688694 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:04.690896 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:46399: BlockingRecv error: recv got EOF from 127.6.68.194:46399 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
May 03 08:15:04 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:04 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:04.818289 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:04.813506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:36351)
0503 08:15:04.813640 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:04.813650 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:04.813708 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:04.813854 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:04.813855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:04.814011 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:04.814169 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:04.814170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.815178 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:04.815179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:04.815545 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:04.815546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.815599 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:04.816225 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:04.816230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:04.818181 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
W20240503 08:15:04.818356 15703 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:04.813551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:36351 (local address 127.6.68.254:43449)
0503 08:15:04.813595 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:04.813597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:04.813719 (+ 122us) server_negotiation.cc:407] Connection header received
0503 08:15:04.813767 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:04.813768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:04.813785 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:04.813851 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:04.814239 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.815098 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:04.815641 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:04.815839 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:04.818291 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:36351: BlockingRecv error: recv got EOF from 127.6.68.195:36351 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
May 03 08:15:05 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:05 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:05.588872 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:05.580453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:53775)
0503 08:15:05.580586 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:05.580597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:05.580652 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:05.580768 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:05.580769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:05.580933 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:05.581104 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:05.581106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.581717 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:05.581718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:05.582072 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:05.582073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.582124 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:05.587119 (+ 4995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:05.587123 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:05.588807 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
W20240503 08:15:05.589026 15707 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:05.580529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:53775 (local address 127.6.68.254:43449)
0503 08:15:05.580647 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:05.580649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:05.580658 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:15:05.580698 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:05.580698 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:05.580709 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:05.580756 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:05.581174 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.581663 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:05.582136 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.582243 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:05.588942 (+ 6699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:53775: BlockingRecv error: recv got EOF from 127.6.68.193:53775 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
May 03 08:15:05 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:05 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:05.695283 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:05.691308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:57089)
0503 08:15:05.691449 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:05.691459 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:05.691510 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:05.691661 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:05.691662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:05.691820 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:05.691999 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:05.692000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.692603 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:05.692603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:05.692963 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:05.692964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.693012 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:05.693576 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:05.693580 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:05.695225 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":56,"threads_started":1}
W20240503 08:15:05.695436 15707 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:05.691437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:57089 (local address 127.6.68.254:43449)
0503 08:15:05.691492 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:05.691493 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:05.691503 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:15:05.691568 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:05.691568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:05.691578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:05.691661 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:05.692055 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.692547 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:05.693015 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.693134 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:05.695312 (+ 2178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:57089: BlockingRecv error: recv got EOF from 127.6.68.194:57089 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
May 03 08:15:05 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:05 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:05.823187 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:05.819012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:50903)
0503 08:15:05.819140 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:05.819154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:05.819218 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:05.819357 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:05.819357 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:05.819535 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:05.819727 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:05.819729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.820341 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:05.820342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:05.820720 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:05.820721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.820784 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:05.821354 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:05.821358 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:05.823121 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20240503 08:15:05.823262 15707 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:05.819120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:50903 (local address 127.6.68.254:43449)
0503 08:15:05.819188 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:05.819190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:05.819218 (+ 28us) server_negotiation.cc:407] Connection header received
0503 08:15:05.819286 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:05.819286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:05.819301 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:05.819348 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:05.819799 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.820269 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:05.820790 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:05.820982 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:05.823206 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:50903: BlockingRecv error: recv got EOF from 127.6.68.195:50903 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
May 03 08:15:06 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:06 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:06.593463 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:06.589451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:33543)
0503 08:15:06.589597 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:06.589607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:06.589655 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:06.589786 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:06.589787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:06.589953 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:06.590125 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:06.590128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.590785 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:06.590785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:06.591171 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:06.591173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.591231 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:06.591791 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:06.591795 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:06.593403 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
W20240503 08:15:06.593592 15711 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:06.589555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:33543 (local address 127.6.68.254:43449)
0503 08:15:06.589673 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:06.589675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:06.589682 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:15:06.589703 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:06.589703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:06.589713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:06.589774 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:06.590193 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.590697 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:06.591250 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.591376 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:06.593506 (+ 2130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:33543: BlockingRecv error: recv got EOF from 127.6.68.193:33543 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":53,"threads_started":1}
May 03 08:15:06 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:06 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:06.700135 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:06.695886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:37395)
0503 08:15:06.696035 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:06.696048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:06.696130 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:06.696272 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:06.696273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:06.696463 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:06.696624 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:06.696626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.697328 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:06.697329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:06.697716 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:06.697718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.697788 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:06.698390 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:06.698393 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:06.700073 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20240503 08:15:06.700315 15711 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:06.695979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:37395 (local address 127.6.68.254:43449)
0503 08:15:06.696039 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:06.696041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:06.696117 (+ 76us) server_negotiation.cc:407] Connection header received
0503 08:15:06.696198 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:06.696199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:06.696212 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:06.696264 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:06.696706 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.697258 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:06.697800 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.697998 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:06.700223 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:37395: BlockingRecv error: recv got EOF from 127.6.68.194:37395 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
May 03 08:15:06 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:06 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.195@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:06.828244 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:06.823794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.195:42165)
0503 08:15:06.823981 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:06.823992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:06.824071 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:06.824276 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:06.824277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:06.824507 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:06.824747 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:06.824749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.825416 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:06.825417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:06.825772 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:06.825773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.825830 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:06.826330 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:06.826334 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:06.828126 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
W20240503 08:15:06.828275 15711 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:06.823955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:42165 (local address 127.6.68.254:43449)
0503 08:15:06.824007 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:06.824010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:06.824057 (+ 47us) server_negotiation.cc:407] Connection header received
0503 08:15:06.824156 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:06.824157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:06.824176 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:06.824252 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:06.824832 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.825336 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:06.825863 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:06.825990 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:06.828217 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.195:42165: BlockingRecv error: recv got EOF from 127.6.68.195:42165 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
May 03 08:15:07 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:07 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.193@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:07.598078 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:07.594085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.193:35057)
0503 08:15:07.594243 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:07.594256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:07.594324 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:07.594456 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:07.594457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:07.594621 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:07.594781 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:07.594782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:07.595402 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:07.595403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:07.595825 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:07.595827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:07.595879 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:07.596383 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:07.596387 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:07.598002 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
W20240503 08:15:07.598184 15715 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:07.594204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:35057 (local address 127.6.68.254:43449)
0503 08:15:07.594333 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:07.594335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:07.594344 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:15:07.594371 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:07.594371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:07.594381 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:07.594440 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:07.594842 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:07.595337 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:07.595905 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:07.596040 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:07.598115 (+ 2075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.193:35057: BlockingRecv error: recv got EOF from 127.6.68.193:35057 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
May 03 08:15:07 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
May 03 08:15:07 dist-test-slave-3g67 krb5kdc[14656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.68.194@KRBTEST.COM for oryx/127.6.68.254@KRBTEST.COM, Server not found in Kerberos database
I20240503 08:15:07.704521 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:07.700655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43449 (local address 127.6.68.194:33749)
0503 08:15:07.700809 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:07.700819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:07.700878 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:07.701019 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:07.701019 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:07.701180 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:07.701339 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:07.701340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:07.701924 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:07.701925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:07.702370 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:07.702373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:07.702432 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:07.702899 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:07.702903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:07.704465 (+ 1562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.68.254:43449: Server oryx/127.6.68.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
W20240503 08:15:07.704623 15715 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:07.700728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:33749 (local address 127.6.68.254:43449)
0503 08:15:07.700787 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:07.700789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:07.700873 (+ 84us) server_negotiation.cc:407] Connection header received
0503 08:15:07.700935 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:07.700935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:07.700945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:07.701000 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:07.701399 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:07.701868 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:07.702445 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:07.702561 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:07.704555 (+ 1994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.68.194:33749: BlockingRecv error: recv got EOF from 127.6.68.194:33749 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:15:07.761086 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 15239
I20240503 08:15:07.765693 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 15391
I20240503 08:15:07.770092 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 15524
I20240503 08:15:07.774379 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 15169
2024-05-03T08:15:07Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17341 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2024-05-03T08:15:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:15:07Z Disabled control of system clock
I20240503 08:15:07.800225 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:35571
--webserver_interface=127.6.68.254
--webserver_port=0
--builtin_ntp_servers=127.6.68.212:35015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:35571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240503 08:15:07.877223 15720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:07.877404 15720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:07.877424 15720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:07.878644 15720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:15:07.878682 15720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:07.878695 15720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:07.878705 15720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:15:07.878728 15720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:15:07.880137 15720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:35571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:35571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:07.880321 15720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:07.880532 15720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:07.884935 15725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:07.885421 15728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:07.886220 15726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:07.887058 15720 server_base.cc:1034] running on GCE node
I20240503 08:15:07.887288 15720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:07.887504 15720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:07.888625 15720 hybrid_clock.cc:648] HybridClock initialized: now 1714724107888618 us; error 28 us; skew 500 ppm
I20240503 08:15:07.889847 15720 webserver.cc:460] Webserver started at http://127.6.68.254:40933/ using document root <none> and password file <none>
I20240503 08:15:07.890013 15720 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:07.890050 15720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:07.890132 15720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:07.891057 15720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "7dcb505f7cae4a2791ecf1f2db0cc6c8"
format_stamp: "Formatted at 2024-05-03 08:15:07 on dist-test-slave-3g67"
server_key: "4d70a90fe23a14e06df03c86b4049506"
server_key_iv: "79cc9b2dd63b85dbecb573efad0790f8"
server_key_version: "encryptionkey@0"
I20240503 08:15:07.891321 15720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "7dcb505f7cae4a2791ecf1f2db0cc6c8"
format_stamp: "Formatted at 2024-05-03 08:15:07 on dist-test-slave-3g67"
server_key: "4d70a90fe23a14e06df03c86b4049506"
server_key_iv: "79cc9b2dd63b85dbecb573efad0790f8"
server_key_version: "encryptionkey@0"
I20240503 08:15:07.892500 15720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:15:07.893138 15734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:07.893294 15720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:15:07.893406 15720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "7dcb505f7cae4a2791ecf1f2db0cc6c8"
format_stamp: "Formatted at 2024-05-03 08:15:07 on dist-test-slave-3g67"
server_key: "4d70a90fe23a14e06df03c86b4049506"
server_key_iv: "79cc9b2dd63b85dbecb573efad0790f8"
server_key_version: "encryptionkey@0"
I20240503 08:15:07.893481 15720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:07.925302 15720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:07.925583 15720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:07.925704 15720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:07.929147 15720 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:35571
I20240503 08:15:07.929205 15786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:35571 every 8 connection(s)
I20240503 08:15:07.929484 15720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:15:07.930080 15787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:07.932555 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8: Bootstrap starting.
I20240503 08:15:07.933086 15787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:07.933331 15787 log.cc:824] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8: Log is configured to *not* fsync() on all Append() calls
I20240503 08:15:07.933951 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8: No bootstrap required, opened a new log
I20240503 08:15:07.933954 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 15720
I20240503 08:15:07.934024 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:15:07.934190 6419 external_mini_cluster.cc:1428] Setting key 675a8325c8103eca47da16ac9e2ebf2c
I20240503 08:15:07.935777 15787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcb505f7cae4a2791ecf1f2db0cc6c8" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35571 } }
I20240503 08:15:07.935925 15787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:07.935953 15787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcb505f7cae4a2791ecf1f2db0cc6c8, State: Initialized, Role: FOLLOWER
I20240503 08:15:07.936098 15787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcb505f7cae4a2791ecf1f2db0cc6c8" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35571 } }
I20240503 08:15:07.936167 15787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:15:07.936187 15787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:15:07.936220 15787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:07.936801 15787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcb505f7cae4a2791ecf1f2db0cc6c8" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35571 } }
I20240503 08:15:07.936897 15787 leader_election.cc:302] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dcb505f7cae4a2791ecf1f2db0cc6c8; no voters:
I20240503 08:15:07.937057 15787 leader_election.cc:288] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:15:07.937275 15792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:07.937301 15787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:15:07.937427 15792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [term 1 LEADER]: Becoming Leader. State: Replica: 7dcb505f7cae4a2791ecf1f2db0cc6c8, State: Running, Role: LEADER
I20240503 08:15:07.937525 15792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcb505f7cae4a2791ecf1f2db0cc6c8" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35571 } }
I20240503 08:15:07.937824 15792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dcb505f7cae4a2791ecf1f2db0cc6c8. Latest consensus state: current_term: 1 leader_uuid: "7dcb505f7cae4a2791ecf1f2db0cc6c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcb505f7cae4a2791ecf1f2db0cc6c8" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35571 } } }
I20240503 08:15:07.937932 15792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [sys.catalog]: This master's current role is: LEADER
I20240503 08:15:07.937834 15793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dcb505f7cae4a2791ecf1f2db0cc6c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcb505f7cae4a2791ecf1f2db0cc6c8" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35571 } } }
I20240503 08:15:07.938098 15793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8 [sys.catalog]: This master's current role is: LEADER
W20240503 08:15:07.939330 15808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240503 08:15:07.939383 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:07.934672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.6.68.254:35571)
0503 08:15:07.934905 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:07.934912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:07.934934 (+ 22us) server_negotiation.cc:407] Connection header received
0503 08:15:07.935758 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:07.935767 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:07.935831 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:07.935986 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:07.936402 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:07.937250 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:07.938197 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:07.938470 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:07.938729 (+ 259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:07.938736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:07.938737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:15:07.938998 (+ 261us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:07.939029 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:15:07.939037 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:07.939100 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:07.939177 (+ 77us) server_negotiation.cc:299] Negotiation successful
0503 08:15:07.939226 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
W20240503 08:15:07.939393 15808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:15:07.939464 15801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:15:07.939589 15801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:15:07.940805 15801 catalog_manager.cc:1347] Generated new cluster ID: 7769764ee8014b2da20f83432710d837
I20240503 08:15:07.940860 15801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:15:07.948657 15801 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:15:07.949220 15801 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:15:07.952735 15801 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7dcb505f7cae4a2791ecf1f2db0cc6c8: Generated new TSK 0
I20240503 08:15:07.952891 15801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:15:07.957806 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35571
--builtin_ntp_servers=127.6.68.212:35015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240503 08:15:08.035218 15811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:08.035403 15811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:08.035425 15811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:08.036751 15811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:08.036794 15811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:08.036818 15811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:15:08.038290 15811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:08.038501 15811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:08.038731 15811 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:08.043462 15819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.044123 15816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.044548 15817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:08.045292 15811 server_base.cc:1034] running on GCE node
I20240503 08:15:08.045480 15811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:08.045712 15811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:08.046869 15811 hybrid_clock.cc:648] HybridClock initialized: now 1714724108046846 us; error 37 us; skew 500 ppm
I20240503 08:15:08.048316 15811 webserver.cc:460] Webserver started at http://127.6.68.193:43085/ using document root <none> and password file <none>
I20240503 08:15:08.048528 15811 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:08.048569 15811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:08.048679 15811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:08.049952 15811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "3ed4fd35d6cb4539bdb5d1a0dfc66583"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "34ea11487d825866dd308e25f03e1932"
server_key_iv: "e03bf4734ebf45a06a03ff2148dcad5f"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.050284 15811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "3ed4fd35d6cb4539bdb5d1a0dfc66583"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "34ea11487d825866dd308e25f03e1932"
server_key_iv: "e03bf4734ebf45a06a03ff2148dcad5f"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.051846 15811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20240503 08:15:08.052641 15825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.052801 15811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:15:08.052868 15811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "3ed4fd35d6cb4539bdb5d1a0dfc66583"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "34ea11487d825866dd308e25f03e1932"
server_key_iv: "e03bf4734ebf45a06a03ff2148dcad5f"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.052935 15811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:08.062320 15811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:08.062561 15811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:08.062661 15811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:08.062855 15811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:08.063139 15811 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:08.063171 15811 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.063205 15811 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:08.063239 15811 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.068435 15811 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:42883
I20240503 08:15:08.068483 15938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:42883 every 8 connection(s)
I20240503 08:15:08.068815 15811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:15:08.072314 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 15811
I20240503 08:15:08.072408 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:15:08.072592 6419 external_mini_cluster.cc:1428] Setting key 1ec03b6257a8724cf71aa40fda143318
I20240503 08:15:08.073556 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.069457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:49295 (local address 127.6.68.254:35571)
0503 08:15:08.069531 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:08.069534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:08.069716 (+ 182us) server_negotiation.cc:407] Connection header received
0503 08:15:08.070149 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:08.070151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:08.070167 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:08.070250 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:08.070688 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.071480 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:08.072044 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.072228 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:08.073234 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:08.073238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:08.073238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:15:08.073243 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:08.073257 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:15:08.073261 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:08.073317 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:08.073408 (+ 91us) server_negotiation.cc:299] Negotiation successful
0503 08:15:08.073445 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240503 08:15:08.073647 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35571
--builtin_ntp_servers=127.6.68.212:35015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20240503 08:15:08.073655 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.069445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:35571 (local address 127.6.68.193:49295)
0503 08:15:08.069612 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:08.069641 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:08.070054 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:08.070300 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:08.070304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:08.070312 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:08.070604 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:08.070606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.071559 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:08.071560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:08.071970 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:08.071972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.072055 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:08.073121 (+ 1066us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:15:08.073122 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:08.073139 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:15:08.073155 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:08.073321 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:08.073326 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:08.073330 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:15:08.073412 (+ 82us) client_negotiation.cc:240] Negotiation successful
0503 08:15:08.073479 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20240503 08:15:08.074882 15939 heartbeater.cc:347] Connected to a master server at 127.6.68.254:35571
I20240503 08:15:08.074976 15939 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:08.075176 15939 heartbeater.cc:511] Master 127.6.68.254:35571 requested a full tablet report, sending...
I20240503 08:15:08.075721 15751 ts_manager.cc:194] Registered new tserver with Master: 3ed4fd35d6cb4539bdb5d1a0dfc66583 (127.6.68.193:42883)
I20240503 08:15:08.076687 15751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.193:49295
W20240503 08:15:08.153154 15942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:08.153339 15942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:08.153370 15942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:08.154642 15942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:08.154687 15942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:08.154723 15942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:15:08.156217 15942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:08.156468 15942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:08.156708 15942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:08.161517 15948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.161998 15947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.162397 15950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:08.163340 15942 server_base.cc:1034] running on GCE node
I20240503 08:15:08.163527 15942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:08.163769 15942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:08.164901 15942 hybrid_clock.cc:648] HybridClock initialized: now 1714724108164893 us; error 28 us; skew 500 ppm
I20240503 08:15:08.166240 15942 webserver.cc:460] Webserver started at http://127.6.68.194:37973/ using document root <none> and password file <none>
I20240503 08:15:08.166450 15942 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:08.166492 15942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:08.166591 15942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:08.167609 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "8f0b8edac7de4e10b73465cdf26e9b96"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "2d4ecbb2b9ae79505e739d07d17c54ab"
server_key_iv: "2987743d12cff4a629bdb6c070e123f9"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.167958 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "8f0b8edac7de4e10b73465cdf26e9b96"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "2d4ecbb2b9ae79505e739d07d17c54ab"
server_key_iv: "2987743d12cff4a629bdb6c070e123f9"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.169158 15942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:15:08.169911 15956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.170082 15942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:15:08.170164 15942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "8f0b8edac7de4e10b73465cdf26e9b96"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "2d4ecbb2b9ae79505e739d07d17c54ab"
server_key_iv: "2987743d12cff4a629bdb6c070e123f9"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.170228 15942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:08.194427 15942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:08.194706 15942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:08.194823 15942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:08.195030 15942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:08.195339 15942 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:08.195394 15942 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.195448 15942 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:08.195479 15942 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.201028 15942 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:38501
I20240503 08:15:08.201114 16069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:38501 every 8 connection(s)
I20240503 08:15:08.201388 15942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:15:08.205768 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.202128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:56401 (local address 127.6.68.254:35571)
0503 08:15:08.202178 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:08.202180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:08.202366 (+ 186us) server_negotiation.cc:407] Connection header received
0503 08:15:08.202821 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:08.202822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:08.202831 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:08.202932 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:08.203420 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.203928 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:08.204440 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.204568 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:08.205494 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:08.205499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:08.205499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:15:08.205504 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:08.205519 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:15:08.205522 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:08.205580 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:08.205673 (+ 93us) server_negotiation.cc:299] Negotiation successful
0503 08:15:08.205707 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240503 08:15:08.205868 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.202091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:35571 (local address 127.6.68.194:56401)
0503 08:15:08.202264 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:08.202292 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:08.202736 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:08.202995 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:08.203000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:08.203011 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:08.203339 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:08.203343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.203990 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:08.203991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:08.204352 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:08.204353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.204447 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:08.205359 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:15:08.205360 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:08.205376 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:15:08.205394 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:08.205584 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:08.205586 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:08.205589 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:15:08.205665 (+ 76us) client_negotiation.cc:240] Negotiation successful
0503 08:15:08.205722 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20240503 08:15:08.206343 16070 heartbeater.cc:347] Connected to a master server at 127.6.68.254:35571
I20240503 08:15:08.206451 16070 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:08.206660 16070 heartbeater.cc:511] Master 127.6.68.254:35571 requested a full tablet report, sending...
I20240503 08:15:08.207047 15751 ts_manager.cc:194] Registered new tserver with Master: 8f0b8edac7de4e10b73465cdf26e9b96 (127.6.68.194:38501)
I20240503 08:15:08.207410 15751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.194:56401
I20240503 08:15:08.208412 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 15942
I20240503 08:15:08.208484 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:15:08.208636 6419 external_mini_cluster.cc:1428] Setting key 0764e1989384537a7459b72dfb567e81
I20240503 08:15:08.209496 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35571
--builtin_ntp_servers=127.6.68.212:35015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240503 08:15:08.288372 16073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:08.288558 16073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:08.288578 16073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:08.289844 16073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:08.289888 16073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:08.289911 16073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:15:08.291328 16073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:08.291523 16073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:08.291769 16073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:08.296222 16079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.296737 16078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.297310 16081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:08.298195 16073 server_base.cc:1034] running on GCE node
I20240503 08:15:08.298375 16073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:08.298599 16073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:08.299731 16073 hybrid_clock.cc:648] HybridClock initialized: now 1714724108299714 us; error 37 us; skew 500 ppm
I20240503 08:15:08.301016 16073 webserver.cc:460] Webserver started at http://127.6.68.195:32789/ using document root <none> and password file <none>
I20240503 08:15:08.301203 16073 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:08.301249 16073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:08.301360 16073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:08.302366 16073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "8bab48d550dc4cc590ae9f034d17e8bf"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "11b79f6b36b0a3521ed760033c7a7f35"
server_key_iv: "1c29b604b43e139adcbf2caf23ad1da3"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.302673 16073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "8bab48d550dc4cc590ae9f034d17e8bf"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "11b79f6b36b0a3521ed760033c7a7f35"
server_key_iv: "1c29b604b43e139adcbf2caf23ad1da3"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.303957 16073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:15:08.304579 16087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.304764 16073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:15:08.304841 16073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "8bab48d550dc4cc590ae9f034d17e8bf"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "11b79f6b36b0a3521ed760033c7a7f35"
server_key_iv: "1c29b604b43e139adcbf2caf23ad1da3"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.304903 16073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:08.316390 16073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:08.316666 16073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:08.316783 16073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:08.317005 16073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:08.317298 16073 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:08.317330 16073 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.317368 16073 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:08.317399 16073 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.322793 16073 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:40235
I20240503 08:15:08.322832 16200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:40235 every 8 connection(s)
I20240503 08:15:08.323124 16073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:15:08.323805 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 16073
I20240503 08:15:08.323896 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:15:08.324106 6419 external_mini_cluster.cc:1428] Setting key 3b9db5411c9a897834fd4a291650551f
I20240503 08:15:08.327951 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.323888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:35339 (local address 127.6.68.254:35571)
0503 08:15:08.324099 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:08.324101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:08.324294 (+ 193us) server_negotiation.cc:407] Connection header received
0503 08:15:08.324998 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:08.325000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:08.325012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:08.325054 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:08.325518 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.325994 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:08.326516 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.326680 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:08.327674 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:08.327678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:08.327678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:15:08.327682 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:08.327699 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:15:08.327703 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:08.327750 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:08.327839 (+ 89us) server_negotiation.cc:299] Negotiation successful
0503 08:15:08.327876 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20240503 08:15:08.328094 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.323840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:35571 (local address 127.6.68.195:35339)
0503 08:15:08.324112 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:08.324171 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:08.324862 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:08.325111 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:08.325117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:08.325126 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:08.325434 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:08.325437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.326080 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:08.326082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:08.326440 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:08.326441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:08.326552 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:08.327521 (+ 969us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:15:08.327522 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:08.327540 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:15:08.327559 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:08.327760 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:08.327764 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:08.327767 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:15:08.327839 (+ 72us) client_negotiation.cc:240] Negotiation successful
0503 08:15:08.327906 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
I20240503 08:15:08.328701 16201 heartbeater.cc:347] Connected to a master server at 127.6.68.254:35571
I20240503 08:15:08.328806 16201 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:08.329049 16201 heartbeater.cc:511] Master 127.6.68.254:35571 requested a full tablet report, sending...
I20240503 08:15:08.329469 15751 ts_manager.cc:194] Registered new tserver with Master: 8bab48d550dc4cc590ae9f034d17e8bf (127.6.68.195:40235)
I20240503 08:15:08.329982 15751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.195:35339
I20240503 08:15:08.336141 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240503 08:15:08.338150 15791 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:08.337666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.6.68.254:35571)
0503 08:15:08.337717 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:08.337720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:08.337819 (+ 99us) server_negotiation.cc:407] Connection header received
0503 08:15:08.337876 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:08.337877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:08.337886 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:08.337944 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:08.338090 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43760: BlockingRecv error: recv got EOF from 127.0.0.1:43760 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20240503 08:15:08.338188 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 15811
I20240503 08:15:08.343698 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 15942
I20240503 08:15:08.347877 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 16073
I20240503 08:15:08.352219 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 15720
2024-05-03T08:15:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (577 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2024-05-03T08:15:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:15:08Z Disabled control of system clock
I20240503 08:15:08.377786 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:36337
--webserver_interface=127.6.68.254
--webserver_port=0
--builtin_ntp_servers=127.6.68.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:36337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240503 08:15:08.456127 16212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:08.456323 16212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:08.456343 16212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:08.457651 16212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:15:08.457697 16212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:08.457710 16212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:08.457722 16212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:15:08.457734 16212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:15:08.459169 16212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:36337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:36337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:08.459435 16212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:08.459702 16212 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:08.464004 16217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.464498 16218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.465025 16220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:08.465996 16212 server_base.cc:1034] running on GCE node
I20240503 08:15:08.466210 16212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:08.466421 16212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:08.467541 16212 hybrid_clock.cc:648] HybridClock initialized: now 1714724108467523 us; error 37 us; skew 500 ppm
I20240503 08:15:08.468720 16212 webserver.cc:460] Webserver started at http://127.6.68.254:41133/ using document root <none> and password file <none>
I20240503 08:15:08.468915 16212 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:08.468958 16212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:08.469063 16212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:08.470078 16212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "e9224d9bb19e4fb79979f7fbf3081a71"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "3113b4c1b475cf085d33d00307456ca4"
server_key_iv: "aaae62955b978d1076a3a72e2cd3cbf1"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.470377 16212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "e9224d9bb19e4fb79979f7fbf3081a71"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "3113b4c1b475cf085d33d00307456ca4"
server_key_iv: "aaae62955b978d1076a3a72e2cd3cbf1"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.471567 16212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:15:08.472249 16226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.472407 16212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:15:08.472483 16212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "e9224d9bb19e4fb79979f7fbf3081a71"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "3113b4c1b475cf085d33d00307456ca4"
server_key_iv: "aaae62955b978d1076a3a72e2cd3cbf1"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.472548 16212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:08.476030 16212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:08.476295 16212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:08.476406 16212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:08.479890 16212 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:36337
I20240503 08:15:08.479920 16278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:36337 every 8 connection(s)
I20240503 08:15:08.480216 16212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:15:08.480775 16279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:08.481463 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 16212
I20240503 08:15:08.481570 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:15:08.481742 6419 external_mini_cluster.cc:1428] Setting key 1b399eeb9e5fe5227719fa292d6f468e
I20240503 08:15:08.483765 16279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71: Bootstrap starting.
I20240503 08:15:08.484252 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.482301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.6.68.254:36337)
0503 08:15:08.482518 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:08.482525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:08.482541 (+ 16us) server_negotiation.cc:407] Connection header received
0503 08:15:08.482922 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:08.482928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:08.482960 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:08.483092 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:08.483907 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:08.483910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:08.483910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:15:08.483914 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:08.483938 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:15:08.483944 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:08.484008 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:08.484084 (+ 76us) server_negotiation.cc:299] Negotiation successful
0503 08:15:08.484129 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20240503 08:15:08.484344 16279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:08.484618 16279 log.cc:824] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71: Log is configured to *not* fsync() on all Append() calls
I20240503 08:15:08.485255 16279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71: No bootstrap required, opened a new log
I20240503 08:15:08.486418 16279 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9224d9bb19e4fb79979f7fbf3081a71" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36337 } }
I20240503 08:15:08.486533 16279 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:08.486562 16279 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9224d9bb19e4fb79979f7fbf3081a71, State: Initialized, Role: FOLLOWER
I20240503 08:15:08.486655 16279 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9224d9bb19e4fb79979f7fbf3081a71" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36337 } }
I20240503 08:15:08.486724 16279 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:15:08.486757 16279 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:15:08.486810 16279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:08.487301 16279 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9224d9bb19e4fb79979f7fbf3081a71" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36337 } }
I20240503 08:15:08.487399 16279 leader_election.cc:302] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9224d9bb19e4fb79979f7fbf3081a71; no voters:
I20240503 08:15:08.487541 16279 leader_election.cc:288] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:15:08.487583 16284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:08.487746 16284 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [term 1 LEADER]: Becoming Leader. State: Replica: e9224d9bb19e4fb79979f7fbf3081a71, State: Running, Role: LEADER
I20240503 08:15:08.487777 16279 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:15:08.487828 16284 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9224d9bb19e4fb79979f7fbf3081a71" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36337 } }
I20240503 08:15:08.488138 16286 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9224d9bb19e4fb79979f7fbf3081a71. Latest consensus state: current_term: 1 leader_uuid: "e9224d9bb19e4fb79979f7fbf3081a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9224d9bb19e4fb79979f7fbf3081a71" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36337 } } }
I20240503 08:15:08.488216 16286 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [sys.catalog]: This master's current role is: LEADER
I20240503 08:15:08.488132 16285 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9224d9bb19e4fb79979f7fbf3081a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9224d9bb19e4fb79979f7fbf3081a71" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 36337 } } }
I20240503 08:15:08.488286 16285 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71 [sys.catalog]: This master's current role is: LEADER
W20240503 08:15:08.489706 16300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:15:08.489929 16300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:15:08.489986 16288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:15:08.490141 16288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:15:08.491410 16288 catalog_manager.cc:1347] Generated new cluster ID: 54084d705d5740bdbd7a00f7efc69988
I20240503 08:15:08.491480 16288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:15:08.513845 16288 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:15:08.514034 16288 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:15:08.516919 16288 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e9224d9bb19e4fb79979f7fbf3081a71: Generated new TSK 0
I20240503 08:15:08.517124 16288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:15:08.525553 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36337
--builtin_ntp_servers=127.6.68.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240503 08:15:08.603782 16303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:08.603999 16303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:08.604033 16303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:08.605317 16303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:08.605360 16303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:08.605399 16303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:15:08.606786 16303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:08.607007 16303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:08.607234 16303 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:08.612485 16308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.613282 16309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.613771 16311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:08.614512 16303 server_base.cc:1034] running on GCE node
I20240503 08:15:08.614674 16303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:08.614889 16303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:08.616048 16303 hybrid_clock.cc:648] HybridClock initialized: now 1714724108616023 us; error 41 us; skew 500 ppm
I20240503 08:15:08.617525 16303 webserver.cc:460] Webserver started at http://127.6.68.193:33813/ using document root <none> and password file <none>
I20240503 08:15:08.617713 16303 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:08.617759 16303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:08.617877 16303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:08.618842 16303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "75b8f0420a644467b8e844bfab8eb506"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "8205fc1ec2f0316bdfb49008a292501a"
server_key_iv: "c5bca1151c68d0bd754ea3407d500924"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.619138 16303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "75b8f0420a644467b8e844bfab8eb506"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "8205fc1ec2f0316bdfb49008a292501a"
server_key_iv: "c5bca1151c68d0bd754ea3407d500924"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.620368 16303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:15:08.621044 16317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.621196 16303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:15:08.621270 16303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "75b8f0420a644467b8e844bfab8eb506"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "8205fc1ec2f0316bdfb49008a292501a"
server_key_iv: "c5bca1151c68d0bd754ea3407d500924"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.621346 16303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:08.624810 16303 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:08.625027 16303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:08.625116 16303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:08.625310 16303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:08.625612 16303 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:08.625651 16303 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.625670 16303 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:08.625684 16303 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.632287 16303 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:40361
I20240503 08:15:08.632318 16430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:40361 every 8 connection(s)
I20240503 08:15:08.632620 16303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:15:08.635535 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.633383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:38063 (local address 127.6.68.254:36337)
0503 08:15:08.633448 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:08.633450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:08.633670 (+ 220us) server_negotiation.cc:407] Connection header received
0503 08:15:08.634321 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:08.634322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:08.634329 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:08.634395 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:08.635296 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:08.635297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:08.635297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:15:08.635301 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:08.635319 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:15:08.635323 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:08.635377 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:08.635466 (+ 89us) server_negotiation.cc:299] Negotiation successful
0503 08:15:08.635478 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240503 08:15:08.635746 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.633311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:36337 (local address 127.6.68.193:38063)
0503 08:15:08.633549 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:08.633577 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:08.634149 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:08.634476 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:08.634480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:08.634490 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:08.635137 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:15:08.635138 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:08.635164 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:15:08.635190 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:08.635387 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:08.635390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:08.635393 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:15:08.635491 (+ 98us) client_negotiation.cc:240] Negotiation successful
0503 08:15:08.635541 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20240503 08:15:08.636391 16431 heartbeater.cc:347] Connected to a master server at 127.6.68.254:36337
I20240503 08:15:08.636489 16431 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:08.636620 16431 heartbeater.cc:511] Master 127.6.68.254:36337 requested a full tablet report, sending...
I20240503 08:15:08.637135 16243 ts_manager.cc:194] Registered new tserver with Master: 75b8f0420a644467b8e844bfab8eb506 (127.6.68.193:40361)
I20240503 08:15:08.640091 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 16303
I20240503 08:15:08.640184 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:15:08.640367 6419 external_mini_cluster.cc:1428] Setting key a82fd634e8da1b41f59eba2288b87a30
I20240503 08:15:08.641350 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36337
--builtin_ntp_servers=127.6.68.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240503 08:15:08.719256 16434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:08.719444 16434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:08.719465 16434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:08.720748 16434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:08.720791 16434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:08.720815 16434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:15:08.722224 16434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:08.722424 16434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:08.722649 16434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:08.727180 16440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.727705 16439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.728363 16442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:08.729205 16434 server_base.cc:1034] running on GCE node
I20240503 08:15:08.729404 16434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:08.729610 16434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:08.730742 16434 hybrid_clock.cc:648] HybridClock initialized: now 1714724108730730 us; error 38 us; skew 500 ppm
I20240503 08:15:08.732044 16434 webserver.cc:460] Webserver started at http://127.6.68.194:44903/ using document root <none> and password file <none>
I20240503 08:15:08.732226 16434 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:08.732270 16434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:08.732403 16434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:08.733384 16434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "554a7d9c0d1145cd8b548b4a13aa8384"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "0bfc0a54b07b9cdccc9c2403d9b1298a"
server_key_iv: "3e559cd8a281c8e504755fd53cbf9a09"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.733676 16434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "554a7d9c0d1145cd8b548b4a13aa8384"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "0bfc0a54b07b9cdccc9c2403d9b1298a"
server_key_iv: "3e559cd8a281c8e504755fd53cbf9a09"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.734862 16434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:15:08.735520 16448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.735710 16434 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:15:08.735787 16434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "554a7d9c0d1145cd8b548b4a13aa8384"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "0bfc0a54b07b9cdccc9c2403d9b1298a"
server_key_iv: "3e559cd8a281c8e504755fd53cbf9a09"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.735852 16434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:08.739463 16434 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:08.739702 16434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:08.739804 16434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:08.739972 16434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:08.740238 16434 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:08.740268 16434 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.740301 16434 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:08.740332 16434 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.745576 16434 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:44745
I20240503 08:15:08.745635 16561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:44745 every 8 connection(s)
I20240503 08:15:08.745888 16434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:15:08.748175 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.746528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:42563 (local address 127.6.68.254:36337)
0503 08:15:08.746587 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:08.746589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:08.746787 (+ 198us) server_negotiation.cc:407] Connection header received
0503 08:15:08.747232 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:08.747233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:08.747238 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:08.747300 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:08.747979 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:08.747980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:08.747980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:15:08.747984 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:08.747997 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:15:08.748000 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:08.748050 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:08.748110 (+ 60us) server_negotiation.cc:299] Negotiation successful
0503 08:15:08.748122 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240503 08:15:08.748303 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.746494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:36337 (local address 127.6.68.194:42563)
0503 08:15:08.746685 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:08.746713 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:08.747141 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:08.747356 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:08.747359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:08.747366 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:08.747863 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:15:08.747863 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:08.747880 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:15:08.747901 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:08.748051 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:08.748053 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:08.748055 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:15:08.748121 (+ 66us) client_negotiation.cc:240] Negotiation successful
0503 08:15:08.748156 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20240503 08:15:08.748893 16562 heartbeater.cc:347] Connected to a master server at 127.6.68.254:36337
I20240503 08:15:08.748966 16562 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:08.749068 16562 heartbeater.cc:511] Master 127.6.68.254:36337 requested a full tablet report, sending...
I20240503 08:15:08.749392 16243 ts_manager.cc:194] Registered new tserver with Master: 554a7d9c0d1145cd8b548b4a13aa8384 (127.6.68.194:44745)
I20240503 08:15:08.755733 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 16434
I20240503 08:15:08.755821 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:15:08.755961 6419 external_mini_cluster.cc:1428] Setting key 21d6207e9a51b6f6e6b60e29f39b03a0
I20240503 08:15:08.756820 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36337
--builtin_ntp_servers=127.6.68.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240503 08:15:08.834275 16565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:08.834443 16565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:08.834463 16565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:08.835752 16565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:08.835789 16565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:08.835813 16565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:15:08.837167 16565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:08.837332 16565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:08.837522 16565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:08.842053 16571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.842720 16573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:08.843394 16570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:08.844092 16565 server_base.cc:1034] running on GCE node
I20240503 08:15:08.844238 16565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:08.844411 16565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:08.845544 16565 hybrid_clock.cc:648] HybridClock initialized: now 1714724108845527 us; error 34 us; skew 500 ppm
I20240503 08:15:08.846776 16565 webserver.cc:460] Webserver started at http://127.6.68.195:43673/ using document root <none> and password file <none>
I20240503 08:15:08.846948 16565 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:08.846984 16565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:08.847070 16565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:08.848053 16565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "9ed736eb2a11447aa024b3a356670aa6"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "9903123ebc4b3254ba46c327081c991c"
server_key_iv: "7a55a0e4f4fe1fdc294b13f42a220bf9"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.848316 16565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "9ed736eb2a11447aa024b3a356670aa6"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "9903123ebc4b3254ba46c327081c991c"
server_key_iv: "7a55a0e4f4fe1fdc294b13f42a220bf9"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.849421 16565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:15:08.850029 16579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.850207 16565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:15:08.850282 16565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "9ed736eb2a11447aa024b3a356670aa6"
format_stamp: "Formatted at 2024-05-03 08:15:08 on dist-test-slave-3g67"
server_key: "9903123ebc4b3254ba46c327081c991c"
server_key_iv: "7a55a0e4f4fe1fdc294b13f42a220bf9"
server_key_version: "encryptionkey@0"
I20240503 08:15:08.850347 16565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:08.853833 16565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:08.854027 16565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:08.854127 16565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:08.854301 16565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:08.854560 16565 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:08.854590 16565 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.854621 16565 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:08.854655 16565 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:08.859961 16565 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:33453
I20240503 08:15:08.859997 16692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:33453 every 8 connection(s)
I20240503 08:15:08.860282 16565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:15:08.860777 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 16565
I20240503 08:15:08.860863 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:15:08.861088 6419 external_mini_cluster.cc:1428] Setting key b32938149661187e906ce90d2236b336
I20240503 08:15:08.862862 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.861042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:47327 (local address 127.6.68.254:36337)
0503 08:15:08.861092 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:08.861094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:08.861366 (+ 272us) server_negotiation.cc:407] Connection header received
0503 08:15:08.861890 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:08.861891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:08.861897 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:08.861941 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:08.862617 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:08.862618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:08.862618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:15:08.862622 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:08.862637 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:15:08.862641 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:08.862698 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:08.862776 (+ 78us) server_negotiation.cc:299] Negotiation successful
0503 08:15:08.862791 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240503 08:15:08.863023 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:08.860944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:36337 (local address 127.6.68.195:47327)
0503 08:15:08.861174 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:08.861206 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:08.861799 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:08.862004 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:08.862008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:08.862017 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:08.862489 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:15:08.862490 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:08.862506 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:15:08.862540 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:08.862702 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:08.862705 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:08.862708 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:15:08.862800 (+ 92us) client_negotiation.cc:240] Negotiation successful
0503 08:15:08.862848 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20240503 08:15:08.863557 16693 heartbeater.cc:347] Connected to a master server at 127.6.68.254:36337
I20240503 08:15:08.863656 16693 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:08.863775 16693 heartbeater.cc:511] Master 127.6.68.254:36337 requested a full tablet report, sending...
I20240503 08:15:08.864091 16243 ts_manager.cc:194] Registered new tserver with Master: 9ed736eb2a11447aa024b3a356670aa6 (127.6.68.195:33453)
I20240503 08:15:08.872771 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240503 08:15:08.874452 16282 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:15:08.874045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.6.68.254:36337)
0503 08:15:08.874103 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:08.874104 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:08.874190 (+ 86us) server_negotiation.cc:407] Connection header received
0503 08:15:08.874231 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:08.874232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:08.874236 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:08.874288 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:08.874386 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43552: BlockingRecv error: recv got EOF from 127.0.0.1:43552 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:15:08.874547 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 16303
I20240503 08:15:08.879244 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 16434
I20240503 08:15:08.883610 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 16565
I20240503 08:15:08.887756 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 16212
2024-05-03T08:15:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (535 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:15:08 dist-test-slave-3g67 krb5kdc[16703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:15:08 dist-test-slave-3g67 krb5kdc[16703](info): set up 2 sockets
May 03 08:15:08 dist-test-slave-3g67 krb5kdc[16703](info): commencing operation
krb5kdc: starting...
W20240503 08:15:09.836369 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:15:09 dist-test-slave-3g67 krb5kdc[16703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:15:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:15:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:15:09.951157 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:37897
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:37897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:15:10.030270 16716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:10.030467 16716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:10.030488 16716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:10.031728 16716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:15:10.031764 16716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:10.031776 16716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:10.031788 16716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:15:10.031800 16716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:15:10.033294 16716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:37897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:37897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:10.033449 16716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:10.033584 16716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:10.037963 16724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:10.038688 16721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:10.039229 16722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:10.039886 16716 server_base.cc:1034] running on GCE node
I20240503 08:15:10.040079 16716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:10.040270 16716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:10.041399 16716 hybrid_clock.cc:648] HybridClock initialized: now 1714724110041387 us; error 30 us; skew 500 ppm
May 03 08:15:10 dist-test-slave-3g67 krb5kdc[16703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724110, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:15:10.043668 16716 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:15:10.044072 16716 webserver.cc:460] Webserver started at http://127.6.68.254:39799/ using document root <none> and password file <none>
I20240503 08:15:10.044238 16716 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:10.044282 16716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:10.044399 16716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:10.045352 16716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "7ff6057848984c8a9581c762be339a4d"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "a305eb9cc225df441fbda1f885838e98"
server_key_iv: "2c092274727a42e5b0a6e7c1294f3ec1"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.045642 16716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "7ff6057848984c8a9581c762be339a4d"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "a305eb9cc225df441fbda1f885838e98"
server_key_iv: "2c092274727a42e5b0a6e7c1294f3ec1"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.046762 16716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:15:10.047377 16731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:10.047529 16716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:15:10.047602 16716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "7ff6057848984c8a9581c762be339a4d"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "a305eb9cc225df441fbda1f885838e98"
server_key_iv: "2c092274727a42e5b0a6e7c1294f3ec1"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.047693 16716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:10.067994 16716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:10.069650 16716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:10.069770 16716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:10.073369 16716 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:37897
I20240503 08:15:10.073416 16783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:37897 every 8 connection(s)
I20240503 08:15:10.073711 16716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:15:10.074476 16784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.075349 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 16716
I20240503 08:15:10.075453 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:15:10.075613 6419 external_mini_cluster.cc:1428] Setting key 892fc1b6e80ff56e35978bd2afa9a4b2
I20240503 08:15:10.077011 16784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d: Bootstrap starting.
I20240503 08:15:10.077549 16784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.077775 16784 log.cc:824] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:15:10.078369 16784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d: No bootstrap required, opened a new log
I20240503 08:15:10.080039 16784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6057848984c8a9581c762be339a4d" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37897 } }
I20240503 08:15:10.080154 16784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.080189 16784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff6057848984c8a9581c762be339a4d, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.080313 16784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6057848984c8a9581c762be339a4d" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37897 } }
I20240503 08:15:10.080385 16784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:15:10.080451 16784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:15:10.080505 16784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [term 0 FOLLOWER]: Advancing to term 1
May 03 08:15:10 dist-test-slave-3g67 krb5kdc[16703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:15:10.081032 16784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6057848984c8a9581c762be339a4d" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37897 } }
I20240503 08:15:10.081115 16784 leader_election.cc:302] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ff6057848984c8a9581c762be339a4d; no voters:
I20240503 08:15:10.081260 16784 leader_election.cc:288] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:15:10.081291 16789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:10.081430 16789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [term 1 LEADER]: Becoming Leader. State: Replica: 7ff6057848984c8a9581c762be339a4d, State: Running, Role: LEADER
I20240503 08:15:10.081496 16784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:15:10.081583 16789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6057848984c8a9581c762be339a4d" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37897 } }
I20240503 08:15:10.081845 16789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ff6057848984c8a9581c762be339a4d. Latest consensus state: current_term: 1 leader_uuid: "7ff6057848984c8a9581c762be339a4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6057848984c8a9581c762be339a4d" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37897 } } }
I20240503 08:15:10.081938 16791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ff6057848984c8a9581c762be339a4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6057848984c8a9581c762be339a4d" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37897 } } }
I20240503 08:15:10.081984 16789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [sys.catalog]: This master's current role is: LEADER
I20240503 08:15:10.082242 16791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d [sys.catalog]: This master's current role is: LEADER
W20240503 08:15:10.083441 16805 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:15:10.083495 16805 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:15:10.083542 16794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:15:10.083698 16794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:15:10.084949 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.076115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.6.68.254:37897)
0503 08:15:10.076293 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:10.076304 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:10.076326 (+ 22us) server_negotiation.cc:407] Connection header received
0503 08:15:10.076849 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:10.076856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:10.076910 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:10.077078 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:10.077774 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.078524 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.079037 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.079264 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.081382 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:10.081390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:10.081390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:15:10.081409 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:10.083737 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:10.084038 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:10.084041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:10.084041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:10.084097 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:10.084255 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:10.084256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:10.084256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:10.084440 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:10.084587 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:10.084681 (+ 94us) server_negotiation.cc:299] Negotiation successful
0503 08:15:10.084798 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20240503 08:15:10.085476 16794 catalog_manager.cc:1347] Generated new cluster ID: 1f9b8de139b94d52853945179cb473ea
I20240503 08:15:10.085525 16794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:15:10.095891 16794 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:15:10.096661 16794 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:15:10.100881 16794 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d: Generated new TSK 0
I20240503 08:15:10.101066 16794 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:15:10.136075 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37897
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:15:10.214236 16810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:10.214445 16810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:10.214479 16810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:10.215819 16810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:10.215867 16810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:10.215906 16810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:15:10.217489 16810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:10.217710 16810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:10.217890 16810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:10.223196 16816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:10.223613 16818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:10.224272 16815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:10.225039 16810 server_base.cc:1034] running on GCE node
I20240503 08:15:10.225196 16810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:10.225392 16810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:10.226526 16810 hybrid_clock.cc:648] HybridClock initialized: now 1714724110226519 us; error 36 us; skew 500 ppm
May 03 08:15:10 dist-test-slave-3g67 krb5kdc[16703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724110, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:15:10.229162 16810 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:15:10.229624 16810 webserver.cc:460] Webserver started at http://127.6.68.193:33125/ using document root <none> and password file <none>
I20240503 08:15:10.229828 16810 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:10.229876 16810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:10.229982 16810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:10.231341 16810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "a4b3e628e9be475fb8cea2dbea55ce70"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "e0cb5c899d48975305be697ee4790791"
server_key_iv: "5f879a54ff347ec238ad5d54cc44160e"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.231791 16810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "a4b3e628e9be475fb8cea2dbea55ce70"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "e0cb5c899d48975305be697ee4790791"
server_key_iv: "5f879a54ff347ec238ad5d54cc44160e"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.233258 16810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:15:10.233978 16825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:10.234146 16810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:15:10.234222 16810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "a4b3e628e9be475fb8cea2dbea55ce70"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "e0cb5c899d48975305be697ee4790791"
server_key_iv: "5f879a54ff347ec238ad5d54cc44160e"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.234279 16810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:10.260752 16810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:10.262836 16810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:10.262954 16810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:10.263139 16810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:10.263444 16810 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:10.263480 16810 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:10.263504 16810 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:10.263532 16810 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:10.269030 16810 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:39649
I20240503 08:15:10.269083 16938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:39649 every 8 connection(s)
I20240503 08:15:10.269376 16810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:15:10.270921 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 16810
I20240503 08:15:10.271029 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:15:10.271198 6419 external_mini_cluster.cc:1428] Setting key cae176a3b762bd792f944354ce532dbb
May 03 08:15:10 dist-test-slave-3g67 krb5kdc[16703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724110, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:15:10.278276 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.270087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:36725 (local address 127.6.68.254:37897)
0503 08:15:10.270135 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:10.270138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:10.270302 (+ 164us) server_negotiation.cc:407] Connection header received
0503 08:15:10.270737 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:10.270738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:10.270750 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:10.270834 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:10.272084 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.272839 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.273444 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.273558 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.275771 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:10.275776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:10.275777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:15:10.275785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:10.277162 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:10.277512 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:10.277513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:10.277514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:10.277536 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:10.277696 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:10.277697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:10.277698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:10.277782 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:10.277839 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:10.278092 (+ 253us) server_negotiation.cc:299] Negotiation successful
0503 08:15:10.278183 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:15:10.278373 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.270053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37897 (local address 127.6.68.193:36725)
0503 08:15:10.270193 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:10.270218 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:10.270654 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:10.270884 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:10.270887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:10.271263 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:10.271997 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.272002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.272917 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.272918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:10.273369 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.273371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.273509 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.274048 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:10.274068 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:10.275684 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:10.277242 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:10.277247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:10.277251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:10.277447 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:10.277589 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:10.277590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:10.277590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:10.277645 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:10.277836 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:10.277841 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:10.277970 (+ 129us) client_negotiation.cc:769] Sending connection context
0503 08:15:10.278065 (+ 95us) client_negotiation.cc:240] Negotiation successful
0503 08:15:10.278195 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
I20240503 08:15:10.278889 16939 heartbeater.cc:347] Connected to a master server at 127.6.68.254:37897
I20240503 08:15:10.278982 16939 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:10.279173 16939 heartbeater.cc:511] Master 127.6.68.254:37897 requested a full tablet report, sending...
I20240503 08:15:10.279704 16748 ts_manager.cc:194] Registered new tserver with Master: a4b3e628e9be475fb8cea2dbea55ce70 (127.6.68.193:39649)
I20240503 08:15:10.280311 16748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:36725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:15:10.296619 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37897
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:15:10.380072 16944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:10.380332 16944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:10.380373 16944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:10.381968 16944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:10.382015 16944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:10.382041 16944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:15:10.383731 16944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:10.383942 16944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:10.384135 16944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:10.388877 16949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:10.389092 16952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:10.389644 16950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:10.390601 16944 server_base.cc:1034] running on GCE node
I20240503 08:15:10.390753 16944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:10.390929 16944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:10.392061 16944 hybrid_clock.cc:648] HybridClock initialized: now 1714724110392050 us; error 29 us; skew 500 ppm
May 03 08:15:10 dist-test-slave-3g67 krb5kdc[16703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724110, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:15:10.394340 16944 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:15:10.394747 16944 webserver.cc:460] Webserver started at http://127.6.68.194:40089/ using document root <none> and password file <none>
I20240503 08:15:10.394924 16944 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:10.394971 16944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:10.395082 16944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:10.396077 16944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "f0e09f24d11544e2a7173344918333dc"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "c1fd15e8e36f55119fa8f4dbcc6281f5"
server_key_iv: "f44f768fe8a8713fb7def79b87bd3639"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.396384 16944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "f0e09f24d11544e2a7173344918333dc"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "c1fd15e8e36f55119fa8f4dbcc6281f5"
server_key_iv: "f44f768fe8a8713fb7def79b87bd3639"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.397544 16944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:15:10.398175 16959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:10.398386 16944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:15:10.398468 16944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "f0e09f24d11544e2a7173344918333dc"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "c1fd15e8e36f55119fa8f4dbcc6281f5"
server_key_iv: "f44f768fe8a8713fb7def79b87bd3639"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.398530 16944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:10.408913 16944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:10.410506 16944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:10.410621 16944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:10.410820 16944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:10.411108 16944 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:10.411140 16944 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:10.411175 16944 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:10.411201 16944 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:10.416502 16944 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:36341
I20240503 08:15:10.416550 17072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:36341 every 8 connection(s)
I20240503 08:15:10.416837 16944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:15:10.420898 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 16944
I20240503 08:15:10.421020 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:15:10.421206 6419 external_mini_cluster.cc:1428] Setting key ebd73fc2c9457f3bb582def1e648abdf
May 03 08:15:10 dist-test-slave-3g67 krb5kdc[16703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724110, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:15:10.425398 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.417461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37897 (local address 127.6.68.194:34917)
0503 08:15:10.417600 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:10.417628 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:10.418070 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:10.418298 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:10.418301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:10.418582 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:10.419105 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.419111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.419787 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.419788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:10.420160 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.420162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.420265 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.420744 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:10.420763 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:10.422371 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:10.424288 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:10.424295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:10.424298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:10.424494 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:10.424645 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:10.424646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:10.424646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:10.424692 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:10.424952 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:10.424955 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:10.425063 (+ 108us) client_negotiation.cc:769] Sending connection context
0503 08:15:10.425152 (+ 89us) client_negotiation.cc:240] Negotiation successful
0503 08:15:10.425256 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":54,"threads_started":1}
I20240503 08:15:10.425395 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.417496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:34917 (local address 127.6.68.254:37897)
0503 08:15:10.417556 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:10.417558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:10.417705 (+ 147us) server_negotiation.cc:407] Connection header received
0503 08:15:10.418153 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:10.418154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:10.418162 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:10.418238 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:10.419196 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.419718 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.420226 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.420327 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.422462 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:10.422465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:10.422465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:15:10.422472 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:10.424195 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:10.424559 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:10.424560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:10.424561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:10.424585 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:10.424747 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:10.424747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:10.424748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:10.424881 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:10.424955 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:10.425188 (+ 233us) server_negotiation.cc:299] Negotiation successful
0503 08:15:10.425280 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:15:10.426059 17073 heartbeater.cc:347] Connected to a master server at 127.6.68.254:37897
I20240503 08:15:10.426172 17073 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:10.426429 17073 heartbeater.cc:511] Master 127.6.68.254:37897 requested a full tablet report, sending...
I20240503 08:15:10.426860 16748 ts_manager.cc:194] Registered new tserver with Master: f0e09f24d11544e2a7173344918333dc (127.6.68.194:36341)
I20240503 08:15:10.427225 16748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:34917
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:15:10.447244 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37897
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:15:10.526325 17078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:10.526521 17078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:10.526556 17078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:10.527855 17078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:10.527897 17078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:10.527935 17078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:15:10.529546 17078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:37897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:10.529875 17078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:10.530086 17078 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:10.534803 17086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:10.535097 17083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:10.535698 17084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:10.536592 17078 server_base.cc:1034] running on GCE node
I20240503 08:15:10.536742 17078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:10.536917 17078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:10.538060 17078 hybrid_clock.cc:648] HybridClock initialized: now 1714724110538029 us; error 44 us; skew 500 ppm
May 03 08:15:10 dist-test-slave-3g67 krb5kdc[16703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724110, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:15:10.540423 17078 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:15:10.540833 17078 webserver.cc:460] Webserver started at http://127.6.68.195:42599/ using document root <none> and password file <none>
I20240503 08:15:10.541026 17078 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:10.541064 17078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:10.541157 17078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:15:10.542086 17078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "2583b7c845694025970fb59b051b4c3e"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "b4fe48447a8db33cfcc60e1884bce8c3"
server_key_iv: "dd4bb5ead72c1588ebd2e25ac6c62740"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.542357 17078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "2583b7c845694025970fb59b051b4c3e"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "b4fe48447a8db33cfcc60e1884bce8c3"
server_key_iv: "dd4bb5ead72c1588ebd2e25ac6c62740"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.543473 17078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:15:10.544150 17093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:10.544319 17078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:10.544386 17078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "2583b7c845694025970fb59b051b4c3e"
format_stamp: "Formatted at 2024-05-03 08:15:10 on dist-test-slave-3g67"
server_key: "b4fe48447a8db33cfcc60e1884bce8c3"
server_key_iv: "dd4bb5ead72c1588ebd2e25ac6c62740"
server_key_version: "encryptionkey@0"
I20240503 08:15:10.544459 17078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:10.569573 17078 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:10.571398 17078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:10.571504 17078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:10.571768 17078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:10.572065 17078 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:10.572099 17078 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:10.572124 17078 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:10.572152 17078 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:10.577247 17078 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:34021
I20240503 08:15:10.577340 17206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:34021 every 8 connection(s)
I20240503 08:15:10.577603 17078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:15:10.581753 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 17078
I20240503 08:15:10.581881 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:15:10.582103 6419 external_mini_cluster.cc:1428] Setting key 9ed4626e50a79916d6ec2432ae96c2e9
May 03 08:15:10 dist-test-slave-3g67 krb5kdc[16703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724110, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:15:10.586344 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.578374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:53143 (local address 127.6.68.254:37897)
0503 08:15:10.578432 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:10.578434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:10.578649 (+ 215us) server_negotiation.cc:407] Connection header received
0503 08:15:10.579129 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:10.579130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:10.579138 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:10.579199 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:10.580249 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.580725 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.581309 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.581419 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.583916 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:10.583920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:10.583920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:15:10.583926 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:10.585304 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:10.585673 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:10.585674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:10.585674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:10.585694 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:10.585842 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:10.585842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:10.585842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:10.585929 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:10.585976 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:10.586193 (+ 217us) server_negotiation.cc:299] Negotiation successful
0503 08:15:10.586275 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240503 08:15:10.586473 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.578357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:37897 (local address 127.6.68.195:53143)
0503 08:15:10.578527 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:10.578558 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:10.579039 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:10.579249 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:10.579253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:10.579564 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:10.580164 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.580170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.580802 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.580803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:10.581228 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.581231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.581376 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.582016 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:10.582032 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:10.583830 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:10.585391 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:10.585397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:10.585400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:10.585609 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:10.585749 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:10.585749 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:10.585750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:10.585799 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:10.585973 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:10.585975 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:10.586084 (+ 109us) client_negotiation.cc:769] Sending connection context
0503 08:15:10.586171 (+ 87us) client_negotiation.cc:240] Negotiation successful
0503 08:15:10.586299 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20240503 08:15:10.586915 17207 heartbeater.cc:347] Connected to a master server at 127.6.68.254:37897
I20240503 08:15:10.587007 17207 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:10.587175 17207 heartbeater.cc:511] Master 127.6.68.254:37897 requested a full tablet report, sending...
I20240503 08:15:10.587535 16748 ts_manager.cc:194] Registered new tserver with Master: 2583b7c845694025970fb59b051b4c3e (127.6.68.195:34021)
I20240503 08:15:10.587908 16748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:53143
I20240503 08:15:10.593772 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:15:10.604257 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.594992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.6.68.254:37897)
0503 08:15:10.595078 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:10.595080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:10.595154 (+ 74us) server_negotiation.cc:407] Connection header received
0503 08:15:10.595237 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:10.595238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:10.595247 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:10.595299 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:10.596067 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.596543 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.597072 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.597250 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.602064 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:10.602068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:10.602068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:15:10.602074 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:10.603204 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:10.603510 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:10.603511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:10.603512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:10.603535 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:10.603715 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:10.603716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:10.603717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:10.603836 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:10.603934 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:10.604062 (+ 128us) server_negotiation.cc:299] Negotiation successful
0503 08:15:10.604144 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240503 08:15:10.605793 16748 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:15:10.607197 16748 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:15:10.612131 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.609341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:36341 (local address 127.0.0.1:57690)
0503 08:15:10.609580 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:10.609609 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:10.609684 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:10.610020 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:10.610023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:10.610025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:15:10.610300 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.610302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.611213 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.611214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:10.611845 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.611846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.611910 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.611953 (+ 43us) client_negotiation.cc:769] Sending connection context
0503 08:15:10.612010 (+ 57us) client_negotiation.cc:240] Negotiation successful
0503 08:15:10.612055 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":105,"threads_started":1}
I20240503 08:15:10.612709 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.609438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.6.68.194:36341)
0503 08:15:10.609850 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:10.609858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:10.609866 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:15:10.609901 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:10.609902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:10.609920 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:10.610003 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:15:10.610390 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.611132 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.611954 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.612557 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.612597 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:10.612625 (+ 28us) server_negotiation.cc:299] Negotiation successful
0503 08:15:10.612654 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":80,"threads_started":1}
I20240503 08:15:10.612932 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.609344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39649 (local address 127.0.0.1:35182)
0503 08:15:10.609700 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:10.609712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:10.609771 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:10.610063 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:10.610064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:10.610067 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:15:10.610302 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.610304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.611726 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.611727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:10.612737 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.612740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.612801 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.612805 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:15:10.612834 (+ 29us) client_negotiation.cc:240] Negotiation successful
0503 08:15:10.612866 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":41,"threads_started":1}
I20240503 08:15:10.613184 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.609347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:34021 (local address 127.0.0.1:34554)
0503 08:15:10.609858 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:10.609865 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:10.609906 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:10.610140 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:10.610141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:10.610143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:15:10.610379 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.610380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.611726 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.611727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:10.612995 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.612997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.613060 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.613063 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:15:10.613097 (+ 34us) client_negotiation.cc:240] Negotiation successful
0503 08:15:10.613130 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":94,"threads_started":1}
I20240503 08:15:10.613250 17007 tablet_service.cc:1467] Processing CreateTablet for tablet 5679bfee11cb4bbc821a11dc7361450a (DEFAULT_TABLE table=test-table [id=08aaec47c94041b2addbead6f252bf38]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.613556 17007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5679bfee11cb4bbc821a11dc7361450a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.613826 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.609569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.6.68.193:39649)
0503 08:15:10.609751 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:10.609758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:10.609770 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:15:10.609872 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:10.609874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:10.609892 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:10.609973 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:15:10.610388 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.611603 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.613110 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.613635 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.613682 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:10.613712 (+ 30us) server_negotiation.cc:299] Negotiation successful
0503 08:15:10.613752 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20240503 08:15:10.613986 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.609562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.6.68.195:34021)
0503 08:15:10.609743 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:10.609751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:10.609917 (+ 166us) server_negotiation.cc:407] Connection header received
0503 08:15:10.609964 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:10.609966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:10.609983 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:10.610062 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:15:10.610448 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.611603 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.613294 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.613790 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.613826 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:10.613857 (+ 31us) server_negotiation.cc:299] Negotiation successful
0503 08:15:10.613902 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
I20240503 08:15:10.614271 16873 tablet_service.cc:1467] Processing CreateTablet for tablet 5679bfee11cb4bbc821a11dc7361450a (DEFAULT_TABLE table=test-table [id=08aaec47c94041b2addbead6f252bf38]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.614544 16873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5679bfee11cb4bbc821a11dc7361450a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.616288 17224 tablet_bootstrap.cc:492] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: Bootstrap starting.
I20240503 08:15:10.616822 17141 tablet_service.cc:1467] Processing CreateTablet for tablet 5679bfee11cb4bbc821a11dc7361450a (DEFAULT_TABLE table=test-table [id=08aaec47c94041b2addbead6f252bf38]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:10.616916 17225 tablet_bootstrap.cc:492] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70: Bootstrap starting.
I20240503 08:15:10.616955 17224 tablet_bootstrap.cc:654] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.617115 17141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5679bfee11cb4bbc821a11dc7361450a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:10.617301 17224 log.cc:824] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: Log is configured to *not* fsync() on all Append() calls
I20240503 08:15:10.617520 17225 tablet_bootstrap.cc:654] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.617823 17225 log.cc:824] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70: Log is configured to *not* fsync() on all Append() calls
I20240503 08:15:10.618075 17224 tablet_bootstrap.cc:492] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: No bootstrap required, opened a new log
I20240503 08:15:10.618153 17224 ts_tablet_manager.cc:1367] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:10.618445 17225 tablet_bootstrap.cc:492] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70: No bootstrap required, opened a new log
I20240503 08:15:10.618635 17225 ts_tablet_manager.cc:1367] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:10.620246 17228 tablet_bootstrap.cc:492] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e: Bootstrap starting.
I20240503 08:15:10.620487 17224 raft_consensus.cc:348] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e09f24d11544e2a7173344918333dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36341 } } peers { permanent_uuid: "2583b7c845694025970fb59b051b4c3e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34021 } } peers { permanent_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39649 } }
I20240503 08:15:10.620604 17224 raft_consensus.cc:374] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.620620 17224 raft_consensus.cc:729] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e09f24d11544e2a7173344918333dc, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.620543 17225 raft_consensus.cc:348] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e09f24d11544e2a7173344918333dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36341 } } peers { permanent_uuid: "2583b7c845694025970fb59b051b4c3e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34021 } } peers { permanent_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39649 } }
I20240503 08:15:10.620669 17225 raft_consensus.cc:374] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.620692 17225 raft_consensus.cc:729] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b3e628e9be475fb8cea2dbea55ce70, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.620699 17224 consensus_queue.cc:260] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e09f24d11544e2a7173344918333dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36341 } } peers { permanent_uuid: "2583b7c845694025970fb59b051b4c3e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34021 } } peers { permanent_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39649 } }
I20240503 08:15:10.620790 17225 consensus_queue.cc:260] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e09f24d11544e2a7173344918333dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36341 } } peers { permanent_uuid: "2583b7c845694025970fb59b051b4c3e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34021 } } peers { permanent_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39649 } }
I20240503 08:15:10.620950 17224 ts_tablet_manager.cc:1398] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:15:10.620992 17073 heartbeater.cc:503] Master 127.6.68.254:37897 was elected leader, sending a full tablet report...
I20240503 08:15:10.621059 17225 ts_tablet_manager.cc:1398] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:10.621204 16939 heartbeater.cc:503] Master 127.6.68.254:37897 was elected leader, sending a full tablet report...
I20240503 08:15:10.621244 17228 tablet_bootstrap.cc:654] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:10.621795 17228 log.cc:824] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:15:10.622779 17228 tablet_bootstrap.cc:492] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e: No bootstrap required, opened a new log
I20240503 08:15:10.622833 17228 ts_tablet_manager.cc:1367] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:15:10.624442 17228 raft_consensus.cc:348] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e09f24d11544e2a7173344918333dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36341 } } peers { permanent_uuid: "2583b7c845694025970fb59b051b4c3e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34021 } } peers { permanent_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39649 } }
I20240503 08:15:10.624557 17228 raft_consensus.cc:374] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:10.624574 17228 raft_consensus.cc:729] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2583b7c845694025970fb59b051b4c3e, State: Initialized, Role: FOLLOWER
I20240503 08:15:10.624672 17228 consensus_queue.cc:260] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e09f24d11544e2a7173344918333dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36341 } } peers { permanent_uuid: "2583b7c845694025970fb59b051b4c3e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34021 } } peers { permanent_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39649 } }
I20240503 08:15:10.624877 17228 ts_tablet_manager.cc:1398] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:10.624929 17207 heartbeater.cc:503] Master 127.6.68.254:37897 was elected leader, sending a full tablet report...
W20240503 08:15:10.667449 17074 tablet.cc:2369] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:15:10.739240 17232 raft_consensus.cc:482] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:10.739362 17232 raft_consensus.cc:504] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e09f24d11544e2a7173344918333dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36341 } } peers { permanent_uuid: "2583b7c845694025970fb59b051b4c3e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34021 } } peers { permanent_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39649 } }
I20240503 08:15:10.739722 17232 leader_election.cc:288] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e09f24d11544e2a7173344918333dc (127.6.68.194:36341), a4b3e628e9be475fb8cea2dbea55ce70 (127.6.68.193:39649)
I20240503 08:15:10.742596 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.739910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39649 (local address 127.6.68.195:43843)
0503 08:15:10.739970 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:10.739983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:10.740031 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:10.740201 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:10.740203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:10.740205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:15:10.740472 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.740474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.741412 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.741413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:10.742347 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.742349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.742430 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.742433 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:15:10.742469 (+ 36us) client_negotiation.cc:240] Negotiation successful
0503 08:15:10.742505 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20240503 08:15:10.742604 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.739918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:36341 (local address 127.6.68.195:59435)
0503 08:15:10.740116 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:10.740124 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:10.740176 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:10.740309 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:10.740309 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:10.740310 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:15:10.740534 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.740535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.741409 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.741409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:10.742350 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:10.742352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.742429 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.742433 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:15:10.742469 (+ 36us) client_negotiation.cc:240] Negotiation successful
0503 08:15:10.742505 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":10,"thread_start_us":118,"threads_started":1}
I20240503 08:15:10.743013 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.740007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:59435 (local address 127.6.68.194:36341)
0503 08:15:10.740054 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:10.740057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:10.740185 (+ 128us) server_negotiation.cc:407] Connection header received
0503 08:15:10.740234 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:10.740234 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:10.740243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:10.740288 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:15:10.740599 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.741311 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.742442 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.742894 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.742900 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:10.742918 (+ 18us) server_negotiation.cc:299] Negotiation successful
0503 08:15:10.742946 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:15:10.743013 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.740012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:43843 (local address 127.6.68.193:39649)
0503 08:15:10.740055 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:10.740057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:10.740068 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:15:10.740098 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:10.740099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:10.740108 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:10.740181 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:15:10.740553 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.741341 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.742435 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.742885 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.742894 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:10.742917 (+ 23us) server_negotiation.cc:299] Negotiation successful
0503 08:15:10.742956 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:15:10.743320 16893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5679bfee11cb4bbc821a11dc7361450a" candidate_uuid: "2583b7c845694025970fb59b051b4c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" is_pre_election: true
I20240503 08:15:10.743341 17027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5679bfee11cb4bbc821a11dc7361450a" candidate_uuid: "2583b7c845694025970fb59b051b4c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e09f24d11544e2a7173344918333dc" is_pre_election: true
I20240503 08:15:10.743463 17027 raft_consensus.cc:2454] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2583b7c845694025970fb59b051b4c3e in term 0.
I20240503 08:15:10.743464 16893 raft_consensus.cc:2454] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2583b7c845694025970fb59b051b4c3e in term 0.
I20240503 08:15:10.743695 17094 leader_election.cc:302] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2583b7c845694025970fb59b051b4c3e, f0e09f24d11544e2a7173344918333dc; no voters:
I20240503 08:15:10.743862 17232 raft_consensus.cc:2781] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:10.743942 17232 raft_consensus.cc:482] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:10.743974 17232 raft_consensus.cc:3037] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.744642 17232 raft_consensus.cc:504] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e09f24d11544e2a7173344918333dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36341 } } peers { permanent_uuid: "2583b7c845694025970fb59b051b4c3e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34021 } } peers { permanent_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39649 } }
I20240503 08:15:10.744771 17232 leader_election.cc:288] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [CANDIDATE]: Term 1 election: Requested vote from peers f0e09f24d11544e2a7173344918333dc (127.6.68.194:36341), a4b3e628e9be475fb8cea2dbea55ce70 (127.6.68.193:39649)
I20240503 08:15:10.744941 16893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5679bfee11cb4bbc821a11dc7361450a" candidate_uuid: "2583b7c845694025970fb59b051b4c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b3e628e9be475fb8cea2dbea55ce70"
I20240503 08:15:10.744994 17027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5679bfee11cb4bbc821a11dc7361450a" candidate_uuid: "2583b7c845694025970fb59b051b4c3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e09f24d11544e2a7173344918333dc"
I20240503 08:15:10.745007 16893 raft_consensus.cc:3037] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.745062 17027 raft_consensus.cc:3037] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:10.745725 16893 raft_consensus.cc:2454] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2583b7c845694025970fb59b051b4c3e in term 1.
I20240503 08:15:10.745728 17027 raft_consensus.cc:2454] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2583b7c845694025970fb59b051b4c3e in term 1.
I20240503 08:15:10.745913 17095 leader_election.cc:302] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2583b7c845694025970fb59b051b4c3e, a4b3e628e9be475fb8cea2dbea55ce70; no voters:
I20240503 08:15:10.746026 17232 raft_consensus.cc:2781] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:10.746186 17232 raft_consensus.cc:686] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 1 LEADER]: Becoming Leader. State: Replica: 2583b7c845694025970fb59b051b4c3e, State: Running, Role: LEADER
I20240503 08:15:10.746277 17232 consensus_queue.cc:237] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e09f24d11544e2a7173344918333dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36341 } } peers { permanent_uuid: "2583b7c845694025970fb59b051b4c3e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34021 } } peers { permanent_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39649 } }
I20240503 08:15:10.747022 16748 catalog_manager.cc:5484] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2583b7c845694025970fb59b051b4c3e (127.6.68.195). New cstate: current_term: 1 leader_uuid: "2583b7c845694025970fb59b051b4c3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e09f24d11544e2a7173344918333dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2583b7c845694025970fb59b051b4c3e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 34021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39649 } health_report { overall_health: UNKNOWN } } }
I20240503 08:15:10.758318 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.756379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.6.68.195:34021)
0503 08:15:10.756438 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:10.756440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:10.756450 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:15:10.756486 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:10.756487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:10.756502 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:10.756568 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:15:10.756801 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.757268 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:10.757825 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:10.757977 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:10.758009 (+ 32us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:15:10.758085 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:15:10.758158 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:10.758217 (+ 59us) server_negotiation.cc:299] Negotiation successful
0503 08:15:10.758239 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240503 08:15:10.759902 16893 raft_consensus.cc:1261] T 5679bfee11cb4bbc821a11dc7361450a P a4b3e628e9be475fb8cea2dbea55ce70 [term 1 FOLLOWER]: Refusing update from remote peer 2583b7c845694025970fb59b051b4c3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:15:10.759902 17027 raft_consensus.cc:1261] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc [term 1 FOLLOWER]: Refusing update from remote peer 2583b7c845694025970fb59b051b4c3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:15:10.760228 17232 consensus_queue.cc:1035] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e09f24d11544e2a7173344918333dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:10.760375 17234 consensus_queue.cc:1035] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b3e628e9be475fb8cea2dbea55ce70" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:10.762881 17238 mvcc.cc:205] Tried to move back new op lower bound from 7023509957669908480 to 7023509957617307648. Current Snapshot: MvccSnapshot[applied={T|T < 7023509957669908480}]
I20240503 08:15:10.764901 16747 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42790:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:15:10.764989 16747 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42790:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:15:10.765689 16747 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 7ff6057848984c8a9581c762be339a4d: Sending DeleteTablet for 3 replicas of tablet 5679bfee11cb4bbc821a11dc7361450a
I20240503 08:15:10.766005 17141 tablet_service.cc:1514] Processing DeleteTablet for tablet 5679bfee11cb4bbc821a11dc7361450a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:15:10 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:34554
I20240503 08:15:10.766115 17007 tablet_service.cc:1514] Processing DeleteTablet for tablet 5679bfee11cb4bbc821a11dc7361450a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:15:10 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:57690
I20240503 08:15:10.766247 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 16810
I20240503 08:15:10.766294 16873 tablet_service.cc:1514] Processing DeleteTablet for tablet 5679bfee11cb4bbc821a11dc7361450a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:15:10 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:35182
I20240503 08:15:10.766382 17247 tablet_replica.cc:330] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e: stopping tablet replica
I20240503 08:15:10.766455 17247 raft_consensus.cc:2229] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:15:10.766532 17247 raft_consensus.cc:2258] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:10.766901 17248 tablet_replica.cc:330] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: stopping tablet replica
I20240503 08:15:10.766986 17248 raft_consensus.cc:2229] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:10.767077 17248 raft_consensus.cc:2258] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:10.767455 17248 ts_tablet_manager.cc:1875] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:15:10.768472 17247 ts_tablet_manager.cc:1875] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:15:10.769196 17248 ts_tablet_manager.cc:1888] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:15:10.769325 17248 log.cc:1196] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/5679bfee11cb4bbc821a11dc7361450a
I20240503 08:15:10.769686 17248 ts_tablet_manager.cc:1909] T 5679bfee11cb4bbc821a11dc7361450a P f0e09f24d11544e2a7173344918333dc: Deleting consensus metadata
I20240503 08:15:10.770063 17247 ts_tablet_manager.cc:1888] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:15:10.770128 17247 log.cc:1196] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/5679bfee11cb4bbc821a11dc7361450a
I20240503 08:15:10.770381 17247 ts_tablet_manager.cc:1909] T 5679bfee11cb4bbc821a11dc7361450a P 2583b7c845694025970fb59b051b4c3e: Deleting consensus metadata
I20240503 08:15:10.770355 16732 catalog_manager.cc:4830] TS f0e09f24d11544e2a7173344918333dc (127.6.68.194:36341): tablet 5679bfee11cb4bbc821a11dc7361450a (table test-table [id=08aaec47c94041b2addbead6f252bf38]) successfully deleted
I20240503 08:15:10.770742 16735 catalog_manager.cc:4830] TS 2583b7c845694025970fb59b051b4c3e (127.6.68.195:34021): tablet 5679bfee11cb4bbc821a11dc7361450a (table test-table [id=08aaec47c94041b2addbead6f252bf38]) successfully deleted
W20240503 08:15:10.774659 16733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:39649 (error 108)
I20240503 08:15:10.775178 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:10.775066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39649 (local address 127.0.0.1:35198)
0503 08:15:10.775116 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:10.775148 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:39649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20240503 08:15:10.775261 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 16944
W20240503 08:15:10.775310 16733 catalog_manager.cc:4573] TS a4b3e628e9be475fb8cea2dbea55ce70 (127.6.68.193:39649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5679bfee11cb4bbc821a11dc7361450a: Network error: Client connection negotiation failed: client connection to 127.6.68.193:39649: connect: Connection refused (error 111)
I20240503 08:15:10.780922 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 17078
I20240503 08:15:10.786185 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 16716
2024-05-03T08:15:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1899 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:15:10 dist-test-slave-3g67 krb5kdc[17252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:15:10 dist-test-slave-3g67 krb5kdc[17252](info): set up 2 sockets
May 03 08:15:10 dist-test-slave-3g67 krb5kdc[17252](info): commencing operation
krb5kdc: starting...
W20240503 08:15:11.736366 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:15:11 dist-test-slave-3g67 krb5kdc[17252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:15:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:15:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/rangeradmin_127.6.68.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/rangeradmin_127.6.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/rangerlookup_127.6.68.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/rangerlookup_127.6.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/HTTP_127.6.68.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/HTTP_127.6.68.212@KRBTEST.COM.keytab.
I20240503 08:15:11.896526 6419 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxRG6Mz/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/postgres -l logfile start
2024-05-03 08:15:14.521 UTC [17282] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-03 08:15:14.521 UTC [17282] LOG: listening on IPv4 address "127.6.68.212", port 49383
2024-05-03 08:15:14.525 UTC [17282] LOG: listening on Unix socket "/tmp/.s.PGSQL.49383"
2024-05-03 08:15:14.547 UTC [17285] LOG: database system was shut down at 2024-05-03 08:15:12 UTC
2024-05-03 08:15:14.551 UTC [17282] LOG: database system is ready to accept connections
I20240503 08:15:15.424156 6419 mini_postgres.cc:96] Postgres bound to 49383
2024-05-03 08:15:15.429 UTC [17293] FATAL: database "slave" does not exist
127.6.68.212:49383 - accepting connections
I20240503 08:15:15.430112 6419 mini_ranger.cc:161] Starting Ranger...
I20240503 08:15:15.443943 6419 mini_ranger.cc:84] Created miniranger Postgres user
I20240503 08:15:15.951666 6419 mini_ranger.cc:87] Created ranger Postgres database
I20240503 08:15:15.951750 6419 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-admin
2024-05-03 08:15:16,163 [I] DB FLAVOR :POSTGRES
2024-05-03 08:15:16,164 [I] --------- Verifying Ranger DB connection ---------
2024-05-03 08:15:16,164 [I] Checking connection..
2024-05-03 08:15:16,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-03 08:15:16,419 [I] Checking connection passed.
2024-05-03 08:15:16,420 [I] --------- Verifying version history table ---------
2024-05-03 08:15:16,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-03 08:15:16,660 [I] Table x_db_version_h does not exist in database ranger
2024-05-03 08:15:16,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-03 08:15:16,888 [I] Table x_db_version_h does not exist in database ranger
2024-05-03 08:15:16,888 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-03 08:15:16,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-03 08:15:17.134 UTC [17357] WARNING: there is no transaction in progress
2024-05-03 08:15:17,140 [I] create_dbversion_catalog.sql file imported successfully
2024-05-03 08:15:17,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-03 08:15:17,395 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-03 08:15:17,395 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-03 08:15:17,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-03 08:15:17,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-03 08:15:17,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3g67.c.gcp-upstream.internal','N') ;"
2024-05-03 08:15:18,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-03 08:15:18,378 [I] Table x_portal_user does not exist in database ranger
2024-05-03 08:15:18,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-03 08:15:18,631 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-03 08:15:18,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-03 08:15:18,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-03 08:15:19,125 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-03 08:15:19,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-03 08:15:20.090 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.108 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.125 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.142 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.159 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.180 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.241 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.249 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.264 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.275 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.286 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.299 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.309 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.318 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.325 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:20.339 UTC [17465] WARNING: there is no transaction in progress
2024-05-03 08:15:21,016 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-03 08:15:21,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-03 08:15:21,261 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-03 08:15:21,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-03 08:15:21,493 [I] Table x_portal_user already exists in database 'ranger'
2024-05-03 08:15:21,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-03 08:15:21,741 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-03 08:15:21,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-03 08:15:21,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-03 08:15:22,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3g67.c.gcp-upstream.internal';"
2024-05-03 08:15:22,449 [I] CORE_DB_SCHEMA import status has been updated
2024-05-03 08:15:22,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-03 08:15:22,699 [I] DB_PATCHES have already been applied
I20240503 08:15:22.705173 6419 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxRG6Mz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240503 08:15:22.705245 6419 mini_ranger.cc:193] Using host: 127.6.68.212
I20240503 08:15:22.707760 6419 mini_ranger.cc:227] Ranger admin URL: http://127.6.68.212:59229
May 03, 2024 8:15:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 03, 2024 8:15:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 03, 2024 8:15:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-admin/ews/webapp
May 03, 2024 8:15:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 03, 2024 8:15:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-admin/ews/webapp] .....
May 03, 2024 8:15:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-admin/ews/webapp].
May 03, 2024 8:15:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20240503 08:15:23.738430 6419 mini_ranger.cc:160] Time spent starting Ranger: real 8.308s user 0.006s sys 0.002s
May 03, 2024 8:15:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 03, 2024 8:15:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.MetricsREST
May 03, 2024 8:15:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 03, 2024 8:15:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 03, 2024 8:15:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240503 08:15:44.246783 6419 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/rangerkms_127.6.68.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/rangerkms_127.6.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.68.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.68.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.68.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/HTTP_127.6.68.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.68.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/HTTP_127.6.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 03 08:15:44 dist-test-slave-3g67 krb5kdc[17252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724144, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20240503 08:15:44.322840 6419 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20240503 08:15:44.332682 6419 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20240503 08:15:45.219148 6419 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20240503 08:15:45.219244 6419 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-kms
I20240503 08:15:45.219267 6419 mini_ranger_kms.cc:227] Using postgres at 127.6.68.212:49383
2024-05-03 08:15:45,389 [I] DB FLAVOR :POSTGRES
2024-05-03 08:15:45,389 [I] --------- Verifying Ranger DB connection ---------
2024-05-03 08:15:45,389 [I] Checking connection
2024-05-03 08:15:45,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2024-05-03 08:15:45,632 [I] connection success
2024-05-03 08:15:45,633 [I] --------- Verifying Ranger DB tables ---------
2024-05-03 08:15:45,633 [I] Verifying table ranger_masterkey in database rangerkms
2024-05-03 08:15:45,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2024-05-03 08:15:45,881 [I] Table ranger_masterkey does not exist in database rangerkms
2024-05-03 08:15:45,882 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-03 08:15:45,882 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2024-05-03 08:15:45,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:49383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2024-05-03 08:15:46,133 [I] kms_core_db_postgres.sql DB schema imported successfully
I20240503 08:15:46.570011 6419 mini_ranger_kms.cc:310] Created kms service
I20240503 08:15:46.769834 6419 mini_ranger_kms.cc:326] Created kudu user
I20240503 08:15:46.814565 6419 mini_ranger_kms.cc:343] Created rangerkms user
May 03 08:15:46 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724144, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:15:47.184675 6419 mini_ranger_kms.cc:384] Added ranger policy
I20240503 08:15:47.184794 6419 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20240503 08:15:47.184814 6419 mini_ranger_kms.cc:240] Using host: 127.6.68.212
I20240503 08:15:47.189013 6419 mini_ranger_kms.cc:276] Ranger KMS PID: 17645
I20240503 08:15:47.189116 6419 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.68.212:60383
May 03, 2024 8:15:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 03, 2024 8:15:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 03, 2024 8:15:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 03, 2024 8:15:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-kms/ews/webapp] .....
May 03, 2024 8:15:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-kms/ews/webapp].
May 03, 2024 8:15:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20240503 08:15:49.238682 6419 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.916s user 0.007s sys 0.147s
I20240503 08:15:49.238808 6419 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20240503 08:15:49.238860 6419 mini_ranger_kms.cc:401] 127.6.68.212:60383/kms/v1/keys
May 03 08:15:49 dist-test-slave-3g67 krb5kdc[17252](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714724149, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.68.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 03 08:15:49 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724149, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.68.212@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
May 03 08:15:50 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724149, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.68.212@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
May 03 08:15:52 dist-test-slave-3g67 krb5kdc[17252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:15:52.399709 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45275
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:45275
--ranger_config_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:60383/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:15:52.478535 17696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:52.478721 17696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:52.478741 17696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:52.480064 17696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:15:52.480110 17696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:52.480124 17696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:52.480135 17696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:15:52.480147 17696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:15:52.481837 17696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.68.212:60383/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:45275
--ranger_config_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:52.482009 17696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:52.482177 17696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:52.487008 17702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:52.487810 17701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:52.488634 17696 server_base.cc:1034] running on GCE node
W20240503 08:15:52.489441 17704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:52.489737 17696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:52.489984 17696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:52.491134 17696 hybrid_clock.cc:648] HybridClock initialized: now 1714724152491116 us; error 42 us; skew 500 ppm
May 03 08:15:52 dist-test-slave-3g67 krb5kdc[17252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724152, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:15:52.499037 17696 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:15:52.499548 17696 webserver.cc:460] Webserver started at http://127.6.68.254:45985/ using document root <none> and password file <none>
I20240503 08:15:52.499780 17696 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:52.499826 17696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:52.499954 17696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 03 08:15:52 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724152, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:15:52.550269 17696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "ad6b03740b904a238786ee77002eda35"
format_stamp: "Formatted at 2024-05-03 08:15:52 on dist-test-slave-3g67"
server_key: "6cd33129a7f4a93681b1922e1ec1e6c1"
server_key_iv: "f9f0514dc5bc7c4cc4016335ecffbfdb"
server_key_version: "kuduclusterkey@0"
I20240503 08:15:52.550662 17696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "ad6b03740b904a238786ee77002eda35"
format_stamp: "Formatted at 2024-05-03 08:15:52 on dist-test-slave-3g67"
server_key: "6cd33129a7f4a93681b1922e1ec1e6c1"
server_key_iv: "f9f0514dc5bc7c4cc4016335ecffbfdb"
server_key_version: "kuduclusterkey@0"
I20240503 08:15:52.551959 17696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:15:52.566253 17711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:52.566450 17696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:15:52.566533 17696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "ad6b03740b904a238786ee77002eda35"
format_stamp: "Formatted at 2024-05-03 08:15:52 on dist-test-slave-3g67"
server_key: "6cd33129a7f4a93681b1922e1ec1e6c1"
server_key_iv: "f9f0514dc5bc7c4cc4016335ecffbfdb"
server_key_version: "kuduclusterkey@0"
I20240503 08:15:52.566633 17696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:52.577112 17696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:52.579041 17696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:52.579164 17696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:52.583750 17696 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:45275
I20240503 08:15:52.583808 17763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:45275 every 8 connection(s)
I20240503 08:15:52.584105 17696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:15:52.584767 17764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:52.584940 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 17696
I20240503 08:15:52.585047 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:15:52.585228 6419 external_mini_cluster.cc:1428] Setting key 46f91b038dde831cab9bb80434ebcceb
I20240503 08:15:52.587551 17764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35: Bootstrap starting.
I20240503 08:15:52.588197 17764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:52.588491 17764 log.cc:824] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35: Log is configured to *not* fsync() on all Append() calls
I20240503 08:15:52.589262 17764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35: No bootstrap required, opened a new log
May 03 08:15:52 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:15:52.591110 17764 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b03740b904a238786ee77002eda35" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45275 } }
I20240503 08:15:52.591256 17764 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:52.591280 17764 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6b03740b904a238786ee77002eda35, State: Initialized, Role: FOLLOWER
I20240503 08:15:52.591440 17764 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b03740b904a238786ee77002eda35" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45275 } }
I20240503 08:15:52.591512 17764 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:15:52.591538 17764 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:15:52.591581 17764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:52.592211 17764 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b03740b904a238786ee77002eda35" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45275 } }
I20240503 08:15:52.592316 17764 leader_election.cc:302] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad6b03740b904a238786ee77002eda35; no voters:
I20240503 08:15:52.592509 17764 leader_election.cc:288] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:15:52.592558 17769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:52.592741 17764 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:15:52.593629 17769 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [term 1 LEADER]: Becoming Leader. State: Replica: ad6b03740b904a238786ee77002eda35, State: Running, Role: LEADER
I20240503 08:15:52.593780 17769 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b03740b904a238786ee77002eda35" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45275 } }
I20240503 08:15:52.594152 17769 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad6b03740b904a238786ee77002eda35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b03740b904a238786ee77002eda35" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45275 } } }
I20240503 08:15:52.594291 17769 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [sys.catalog]: This master's current role is: LEADER
I20240503 08:15:52.594337 17764 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240503 08:15:52.594478 17769 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad6b03740b904a238786ee77002eda35. Latest consensus state: current_term: 1 leader_uuid: "ad6b03740b904a238786ee77002eda35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6b03740b904a238786ee77002eda35" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45275 } } }
I20240503 08:15:52.594693 17769 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35 [sys.catalog]: This master's current role is: LEADER
I20240503 08:15:52.596710 17783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:15:52.597786 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:52.585714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.6.68.254:45275)
0503 08:15:52.585893 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:52.585905 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:52.585926 (+ 21us) server_negotiation.cc:407] Connection header received
0503 08:15:52.586463 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:52.586470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:52.586529 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:52.586677 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:52.587485 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:52.588441 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:52.589166 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:52.589440 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:52.591686 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:52.591693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:52.591693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:15:52.591707 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:52.593513 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:52.593880 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:52.593883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:52.593883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:52.593932 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:52.594096 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:52.594097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:52.594098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:52.594332 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:52.594493 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:52.596579 (+ 2086us) server_negotiation.cc:299] Negotiation successful
0503 08:15:52.597498 (+ 919us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20240503 08:15:52.597831 17783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:15:52.608772 17783 catalog_manager.cc:1347] Generated new cluster ID: d5ed7e280ec54b8390222a305d847885
I20240503 08:15:52.608837 17783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:15:52.625854 17783 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:15:52.626688 17783 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:15:52.643780 17783 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35: Generated new TSK 0
I20240503 08:15:52.644063 17783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:15:53.315179 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:60383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45275
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:15:53.412395 17809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:53.412633 17809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:53.412672 17809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:53.414723 17809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:53.414788 17809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:53.414827 17809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:15:53.417120 17809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.68.212:60383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:53.417351 17809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:53.417553 17809 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:53.422760 17815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:53.423388 17814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:53.424408 17817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:53.425091 17809 server_base.cc:1034] running on GCE node
I20240503 08:15:53.425287 17809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:53.425513 17809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:53.426647 17809 hybrid_clock.cc:648] HybridClock initialized: now 1714724153426628 us; error 38 us; skew 500 ppm
May 03 08:15:53 dist-test-slave-3g67 krb5kdc[17252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:15:53.429332 17809 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:15:53.429832 17809 webserver.cc:460] Webserver started at http://127.6.68.193:39869/ using document root <none> and password file <none>
I20240503 08:15:53.430038 17809 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:53.430102 17809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:53.430269 17809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 03 08:15:53 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:15:53.444363 17809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "b439da89a2bf4ad4870914acb7c79497"
format_stamp: "Formatted at 2024-05-03 08:15:53 on dist-test-slave-3g67"
server_key: "51748caf1d6edee3e14e81afab2534f5"
server_key_iv: "3e79ce9c5eca32f59b5bb222f026d0fa"
server_key_version: "kuduclusterkey@0"
I20240503 08:15:53.444824 17809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "b439da89a2bf4ad4870914acb7c79497"
format_stamp: "Formatted at 2024-05-03 08:15:53 on dist-test-slave-3g67"
server_key: "51748caf1d6edee3e14e81afab2534f5"
server_key_iv: "3e79ce9c5eca32f59b5bb222f026d0fa"
server_key_version: "kuduclusterkey@0"
I20240503 08:15:53.446498 17809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:15:53.458928 17824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:53.459151 17809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:15:53.459230 17809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "b439da89a2bf4ad4870914acb7c79497"
format_stamp: "Formatted at 2024-05-03 08:15:53 on dist-test-slave-3g67"
server_key: "51748caf1d6edee3e14e81afab2534f5"
server_key_iv: "3e79ce9c5eca32f59b5bb222f026d0fa"
server_key_version: "kuduclusterkey@0"
I20240503 08:15:53.459316 17809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:53.486306 17809 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:53.488862 17809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:53.489041 17809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:53.489324 17809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:53.489727 17809 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:53.489822 17809 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:53.489929 17809 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:53.490008 17809 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:53.497092 17809 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:33051
I20240503 08:15:53.497562 17809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:15:53.498178 17939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:33051 every 8 connection(s)
I20240503 08:15:53.502065 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 17809
I20240503 08:15:53.502169 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:15:53.502336 6419 external_mini_cluster.cc:1428] Setting key 7b5ea6853744f4c9cb64ab85810f1edf
May 03 08:15:53 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:15:53.512279 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.498876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45275 (local address 127.6.68.193:39523)
0503 08:15:53.500776 (+ 1900us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:53.500810 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:53.501969 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:53.502452 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:53.502457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:53.502722 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:53.503155 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:53.503162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.504247 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:53.504249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:53.504849 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:53.504852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.505003 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:53.505667 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:53.505689 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:53.507340 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:53.510826 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:53.510832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:53.510835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:53.511111 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:53.511301 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:53.511301 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:53.511302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:53.511393 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:53.511688 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:53.511692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:53.511843 (+ 151us) client_negotiation.cc:769] Sending connection context
0503 08:15:53.511946 (+ 103us) client_negotiation.cc:240] Negotiation successful
0503 08:15:53.512071 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1836,"thread_start_us":69,"threads_started":1}
I20240503 08:15:53.513077 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.499146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:39523 (local address 127.6.68.254:45275)
0503 08:15:53.502183 (+ 3037us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:53.502185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:53.502198 (+ 13us) server_negotiation.cc:407] Connection header received
0503 08:15:53.502226 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:53.502227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:53.502243 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:53.502318 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:53.503311 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.504140 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:53.504946 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.505131 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:53.507458 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:53.507463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:53.507463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:15:53.507470 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:53.509335 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:53.511202 (+ 1867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:53.511203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:53.511204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:53.511232 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:53.511466 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:53.511466 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:53.511467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:53.511583 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:53.512829 (+ 1246us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:53.512875 (+ 46us) server_negotiation.cc:299] Negotiation successful
0503 08:15:53.512979 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3004,"thread_start_us":91,"threads_started":1}
I20240503 08:15:53.513622 17940 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45275
I20240503 08:15:53.513758 17940 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:53.514014 17940 heartbeater.cc:511] Master 127.6.68.254:45275 requested a full tablet report, sending...
I20240503 08:15:53.514722 17728 ts_manager.cc:194] Registered new tserver with Master: b439da89a2bf4ad4870914acb7c79497 (127.6.68.193:33051)
I20240503 08:15:53.515736 17728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:39523
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:15:53.543442 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:60383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45275
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:15:53.649406 17946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:53.649590 17946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:53.649621 17946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:53.650892 17946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:53.650934 17946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:53.650972 17946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:15:53.652519 17946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.68.212:60383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:53.652727 17946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:53.652937 17946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:53.657670 17953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:53.658100 17956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:53.659008 17954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:53.659904 17946 server_base.cc:1034] running on GCE node
I20240503 08:15:53.660094 17946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:53.660324 17946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:53.661490 17946 hybrid_clock.cc:648] HybridClock initialized: now 1714724153661461 us; error 47 us; skew 500 ppm
May 03 08:15:53 dist-test-slave-3g67 krb5kdc[17252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:15:53.664710 17946 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:15:53.665205 17946 webserver.cc:460] Webserver started at http://127.6.68.194:37573/ using document root <none> and password file <none>
I20240503 08:15:53.665434 17946 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:53.665499 17946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:53.665663 17946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 03 08:15:53 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:15:53.681540 17946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "c25dfd08db4a4e99b6c3c261f76015dc"
format_stamp: "Formatted at 2024-05-03 08:15:53 on dist-test-slave-3g67"
server_key: "341723ab787af3052aca0914607b5a0e"
server_key_iv: "034db27977eed02c2ff78bb23a5c898d"
server_key_version: "kuduclusterkey@0"
I20240503 08:15:53.681929 17946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "c25dfd08db4a4e99b6c3c261f76015dc"
format_stamp: "Formatted at 2024-05-03 08:15:53 on dist-test-slave-3g67"
server_key: "341723ab787af3052aca0914607b5a0e"
server_key_iv: "034db27977eed02c2ff78bb23a5c898d"
server_key_version: "kuduclusterkey@0"
May 03 08:15:53 dist-test-slave-3g67 krb5kdc[17252](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:15:53.683215 17946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:15:53.698032 17963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:53.698199 17946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:15:53.698279 17946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "c25dfd08db4a4e99b6c3c261f76015dc"
format_stamp: "Formatted at 2024-05-03 08:15:53 on dist-test-slave-3g67"
server_key: "341723ab787af3052aca0914607b5a0e"
server_key_iv: "034db27977eed02c2ff78bb23a5c898d"
server_key_version: "kuduclusterkey@0"
I20240503 08:15:53.698369 17946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:53.726984 17946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:53.728982 17946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:53.729106 17946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:53.729327 17946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:53.729736 17946 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:53.729822 17946 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:53.729907 17946 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:53.729976 17946 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:53.736903 17946 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:34661
I20240503 08:15:53.736951 18077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:34661 every 8 connection(s)
I20240503 08:15:53.737290 17946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:15:53.742185 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 17946
I20240503 08:15:53.742291 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
May 03 08:15:53 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:15:53.742491 6419 external_mini_cluster.cc:1428] Setting key 1e3d09815250d92f00e0233e4a517024
I20240503 08:15:53.745704 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.737966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:46581 (local address 127.6.68.254:45275)
0503 08:15:53.738037 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:53.738039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:53.738211 (+ 172us) server_negotiation.cc:407] Connection header received
0503 08:15:53.738983 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:53.738984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:53.739002 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:53.739059 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:53.739720 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.740221 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:53.740738 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.740937 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:53.742821 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:53.742825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:53.742825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:15:53.742831 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:53.744380 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:53.744732 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:53.744734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:53.744734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:53.744759 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:53.744930 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:53.744931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:53.744932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:53.745048 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:53.745116 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:53.745489 (+ 373us) server_negotiation.cc:299] Negotiation successful
0503 08:15:53.745599 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240503 08:15:53.745764 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.737915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45275 (local address 127.6.68.194:46581)
0503 08:15:53.738120 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:53.738141 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:53.738879 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:53.739125 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:53.739128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:53.739306 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:53.739587 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:53.739591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.740304 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:53.740305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:53.740664 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:53.740666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.740793 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:53.741356 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:53.741371 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:53.742726 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:53.744462 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:53.744468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:53.744470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:53.744667 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:53.744815 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:53.744816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:53.744816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:53.744876 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:53.745148 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:53.745154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:53.745307 (+ 153us) client_negotiation.cc:769] Sending connection context
0503 08:15:53.745430 (+ 123us) client_negotiation.cc:240] Negotiation successful
0503 08:15:53.745568 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":64,"threads_started":1}
I20240503 08:15:53.746184 18078 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45275
I20240503 08:15:53.746280 18078 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:53.746475 18078 heartbeater.cc:511] Master 127.6.68.254:45275 requested a full tablet report, sending...
I20240503 08:15:53.746863 17728 ts_manager.cc:194] Registered new tserver with Master: c25dfd08db4a4e99b6c3c261f76015dc (127.6.68.194:34661)
I20240503 08:15:53.747259 17728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:46581
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:15:53.770591 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:60383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45275
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:15:53.859148 18084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:15:53.859342 18084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:15:53.859373 18084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:15:53.860762 18084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:15:53.860822 18084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:15:53.860862 18084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:15:53.862491 18084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.68.212:60383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:15:53.862715 18084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:15:53.862916 18084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:15:53.867985 18092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:53.868772 18090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:15:53.869504 18089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:15:53.870349 18084 server_base.cc:1034] running on GCE node
I20240503 08:15:53.870522 18084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:15:53.870726 18084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:15:53.871858 18084 hybrid_clock.cc:648] HybridClock initialized: now 1714724153871853 us; error 31 us; skew 500 ppm
May 03 08:15:53 dist-test-slave-3g67 krb5kdc[17252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:15:53.874341 18084 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:15:53.874894 18084 webserver.cc:460] Webserver started at http://127.6.68.195:37233/ using document root <none> and password file <none>
I20240503 08:15:53.875110 18084 fs_manager.cc:359] Metadata directory not provided
I20240503 08:15:53.875180 18084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:15:53.875305 18084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 03 08:15:53 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:15:53.887734 18084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "0227d75f2082486fa170a668266da0d3"
format_stamp: "Formatted at 2024-05-03 08:15:53 on dist-test-slave-3g67"
server_key: "80bbba2bdb0fe93837d13ed48ccf52f9"
server_key_iv: "24c0641cba5bf388b9fb0825d21828ca"
server_key_version: "kuduclusterkey@0"
I20240503 08:15:53.888062 18084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "0227d75f2082486fa170a668266da0d3"
format_stamp: "Formatted at 2024-05-03 08:15:53 on dist-test-slave-3g67"
server_key: "80bbba2bdb0fe93837d13ed48ccf52f9"
server_key_iv: "24c0641cba5bf388b9fb0825d21828ca"
server_key_version: "kuduclusterkey@0"
I20240503 08:15:53.889299 18084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:15:53.908128 18099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:53.908483 18084 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20240503 08:15:53.908658 18084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "0227d75f2082486fa170a668266da0d3"
format_stamp: "Formatted at 2024-05-03 08:15:53 on dist-test-slave-3g67"
server_key: "80bbba2bdb0fe93837d13ed48ccf52f9"
server_key_iv: "24c0641cba5bf388b9fb0825d21828ca"
server_key_version: "kuduclusterkey@0"
I20240503 08:15:53.908828 18084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:15:53.931044 18084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:15:53.933475 18084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:15:53.933660 18084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:15:53.933933 18084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:15:53.934348 18084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:15:53.934435 18084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:53.934504 18084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:15:53.934558 18084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:15:53.941368 18084 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:36347
I20240503 08:15:53.941793 18084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:15:53.943809 18212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:36347 every 8 connection(s)
I20240503 08:15:53.946173 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 18084
I20240503 08:15:53.946283 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:15:53.946449 6419 external_mini_cluster.cc:1428] Setting key aa919001f125c3121dfb14fea6e578d3
May 03 08:15:53 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:15:53.952785 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.942795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:45195 (local address 127.6.68.254:45275)
0503 08:15:53.942838 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:53.942841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:53.942852 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:15:53.944029 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:53.944031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:53.944046 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:53.944076 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:53.945040 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.945838 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:53.946589 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.946758 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:53.949336 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:53.949340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:53.949341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:15:53.949348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:53.951177 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:53.951726 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:53.951728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:53.951728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:53.951755 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:53.952011 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:53.952013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:53.952013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:53.952128 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:53.952195 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:53.952594 (+ 399us) server_negotiation.cc:299] Negotiation successful
0503 08:15:53.952691 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:15:53.952910 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.942533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45275 (local address 127.6.68.195:45195)
0503 08:15:53.942705 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:53.942740 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:53.943833 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:53.944255 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:53.944260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:53.944501 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:15:53.944920 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:53.944926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.945945 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:53.945946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:53.946485 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:53.946487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.946887 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:53.947478 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:15:53.947501 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:15:53.949219 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:15:53.951281 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:53.951288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:53.951292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:53.951647 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:53.951828 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:15:53.951829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:15:53.951830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:15:53.951934 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:15:53.952196 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:15:53.952201 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:15:53.952378 (+ 177us) client_negotiation.cc:769] Sending connection context
0503 08:15:53.952515 (+ 137us) client_negotiation.cc:240] Negotiation successful
0503 08:15:53.952667 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20240503 08:15:53.953433 18213 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45275
I20240503 08:15:53.953548 18213 heartbeater.cc:464] Registering TS with master...
I20240503 08:15:53.953799 18213 heartbeater.cc:511] Master 127.6.68.254:45275 requested a full tablet report, sending...
I20240503 08:15:53.954262 17728 ts_manager.cc:194] Registered new tserver with Master: 0227d75f2082486fa170a668266da0d3 (127.6.68.195:36347)
I20240503 08:15:53.954756 17728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:45195
I20240503 08:15:53.963306 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:15:53.983942 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.976065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38854 (local address 127.6.68.254:45275)
0503 08:15:53.976111 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:53.976114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:53.976444 (+ 330us) server_negotiation.cc:407] Connection header received
0503 08:15:53.976559 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:53.976560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:53.976572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:53.976601 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:15:53.977675 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.978466 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:53.979318 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.979488 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:53.981035 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:15:53.981039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:15:53.981040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:15:53.981046 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:15:53.982778 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:53.983156 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:53.983157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:53.983157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:53.983180 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:15:53.983338 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:15:53.983339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:15:53.983340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:15:53.983443 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:15:53.983515 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:53.983737 (+ 222us) server_negotiation.cc:299] Negotiation successful
0503 08:15:53.983823 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:15:53.986308 17728 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:15:53.988384 17728 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:15:53.996230 18226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.991979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:34661 (local address 127.0.0.1:55284)
0503 08:15:53.993098 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:53.993106 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:53.993150 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:53.993426 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:53.993429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:53.993431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:15:53.993682 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:53.993684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.994992 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:53.994993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:53.995945 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:53.995946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.996010 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:53.996045 (+ 35us) client_negotiation.cc:769] Sending connection context
0503 08:15:53.996100 (+ 55us) client_negotiation.cc:240] Negotiation successful
0503 08:15:53.996150 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"thread_start_us":51,"threads_started":1}
I20240503 08:15:53.997182 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.992233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.6.68.194:34661)
0503 08:15:53.993051 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:53.993060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:53.993197 (+ 137us) server_negotiation.cc:407] Connection header received
0503 08:15:53.993237 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:53.993239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:53.993264 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:53.993333 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:15:53.993769 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.994909 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:53.996379 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.996979 (+ 600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:53.997034 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:53.997064 (+ 30us) server_negotiation.cc:299] Negotiation successful
0503 08:15:53.997107 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":86,"threads_started":1}
I20240503 08:15:53.997864 17993 tablet_service.cc:1467] Processing CreateTablet for tablet 96a2f6405dcf4aa89ae79334b4346467 (DEFAULT_TABLE table=test-table [id=28d5bac7dead4056aa20a4fcfa48d84d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:53.998165 17993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a2f6405dcf4aa89ae79334b4346467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:54.000422 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.991577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:33051 (local address 127.0.0.1:44698)
0503 08:15:53.992697 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:53.992724 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:53.992788 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:53.995746 (+ 2958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:53.995747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:53.995748 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:15:53.995949 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:53.995949 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.999019 (+ 3070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:53.999021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:54.000233 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:54.000236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.000286 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:54.000291 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:15:54.000314 (+ 23us) client_negotiation.cc:240] Negotiation successful
0503 08:15:54.000350 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"thread_start_us":97,"threads_started":1}
I20240503 08:15:54.001695 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.991819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:36347 (local address 127.0.0.1:39762)
0503 08:15:53.992894 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:53.992902 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:53.992940 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:53.995437 (+ 2497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:53.995438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:53.995440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:15:53.995664 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:53.995666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.000598 (+ 4932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:54.000599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:54.001478 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:54.001481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.001523 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:54.001526 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:15:54.001549 (+ 23us) client_negotiation.cc:240] Negotiation successful
0503 08:15:54.001587 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":64,"threads_started":1}
I20240503 08:15:54.001986 18232 tablet_bootstrap.cc:492] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: Bootstrap starting.
I20240503 08:15:54.002691 18232 tablet_bootstrap.cc:654] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:54.003320 18232 log.cc:824] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: Log is configured to *not* fsync() on all Append() calls
I20240503 08:15:54.003935 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.992317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.6.68.195:36347)
0503 08:15:53.995281 (+ 2964us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:53.995288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:53.995296 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:15:53.995328 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:53.995329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:53.995345 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:53.996027 (+ 682us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:15:53.996101 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.998910 (+ 2809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:54.003116 (+ 4206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.003711 (+ 595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:54.003757 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:54.003790 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:15:54.003841 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2932,"thread_start_us":83,"threads_started":1}
I20240503 08:15:54.004132 18232 tablet_bootstrap.cc:492] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: No bootstrap required, opened a new log
I20240503 08:15:54.004191 18232 ts_tablet_manager.cc:1367] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:54.004202 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:53.991677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.6.68.193:33051)
0503 08:15:53.995077 (+ 3400us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:53.995090 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:53.995099 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:15:53.995136 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:53.995138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:53.995154 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:53.996913 (+ 1759us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:15:53.996991 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:53.997992 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:54.001972 (+ 3980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.002510 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:54.002557 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:54.002591 (+ 34us) server_negotiation.cc:299] Negotiation successful
0503 08:15:54.002643 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3365,"thread_start_us":50,"threads_started":1}
I20240503 08:15:54.004724 17855 tablet_service.cc:1467] Processing CreateTablet for tablet 96a2f6405dcf4aa89ae79334b4346467 (DEFAULT_TABLE table=test-table [id=28d5bac7dead4056aa20a4fcfa48d84d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:54.005018 17855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a2f6405dcf4aa89ae79334b4346467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:54.005970 18232 raft_consensus.cc:348] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b439da89a2bf4ad4870914acb7c79497" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33051 } } peers { permanent_uuid: "0227d75f2082486fa170a668266da0d3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36347 } } peers { permanent_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34661 } }
I20240503 08:15:54.006103 18232 raft_consensus.cc:374] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:54.006127 18232 raft_consensus.cc:729] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25dfd08db4a4e99b6c3c261f76015dc, State: Initialized, Role: FOLLOWER
I20240503 08:15:54.006220 18232 consensus_queue.cc:260] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b439da89a2bf4ad4870914acb7c79497" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33051 } } peers { permanent_uuid: "0227d75f2082486fa170a668266da0d3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36347 } } peers { permanent_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34661 } }
I20240503 08:15:54.006466 18232 ts_tablet_manager.cc:1398] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:15:54.006599 18078 heartbeater.cc:503] Master 127.6.68.254:45275 was elected leader, sending a full tablet report...
I20240503 08:15:54.008666 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 96a2f6405dcf4aa89ae79334b4346467 (DEFAULT_TABLE table=test-table [id=28d5bac7dead4056aa20a4fcfa48d84d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:15:54.008919 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a2f6405dcf4aa89ae79334b4346467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:15:54.011023 18236 tablet_bootstrap.cc:492] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: Bootstrap starting.
I20240503 08:15:54.011272 18235 tablet_bootstrap.cc:492] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497: Bootstrap starting.
I20240503 08:15:54.011695 18236 tablet_bootstrap.cc:654] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:54.011907 18235 tablet_bootstrap.cc:654] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497: Neither blocks nor log segments found. Creating new log.
I20240503 08:15:54.012337 18236 log.cc:824] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: Log is configured to *not* fsync() on all Append() calls
I20240503 08:15:54.012537 18235 log.cc:824] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497: Log is configured to *not* fsync() on all Append() calls
I20240503 08:15:54.013211 18235 tablet_bootstrap.cc:492] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497: No bootstrap required, opened a new log
I20240503 08:15:54.013206 18236 tablet_bootstrap.cc:492] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: No bootstrap required, opened a new log
I20240503 08:15:54.013274 18235 ts_tablet_manager.cc:1367] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:15:54.013274 18236 ts_tablet_manager.cc:1367] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:15:54.015018 18236 raft_consensus.cc:348] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b439da89a2bf4ad4870914acb7c79497" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33051 } } peers { permanent_uuid: "0227d75f2082486fa170a668266da0d3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36347 } } peers { permanent_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34661 } }
I20240503 08:15:54.015018 18235 raft_consensus.cc:348] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b439da89a2bf4ad4870914acb7c79497" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33051 } } peers { permanent_uuid: "0227d75f2082486fa170a668266da0d3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36347 } } peers { permanent_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34661 } }
I20240503 08:15:54.015147 18235 raft_consensus.cc:374] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:54.015147 18236 raft_consensus.cc:374] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:15:54.015172 18236 raft_consensus.cc:729] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0227d75f2082486fa170a668266da0d3, State: Initialized, Role: FOLLOWER
I20240503 08:15:54.015172 18235 raft_consensus.cc:729] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b439da89a2bf4ad4870914acb7c79497, State: Initialized, Role: FOLLOWER
I20240503 08:15:54.015255 18235 consensus_queue.cc:260] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b439da89a2bf4ad4870914acb7c79497" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33051 } } peers { permanent_uuid: "0227d75f2082486fa170a668266da0d3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36347 } } peers { permanent_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34661 } }
I20240503 08:15:54.015254 18236 consensus_queue.cc:260] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b439da89a2bf4ad4870914acb7c79497" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33051 } } peers { permanent_uuid: "0227d75f2082486fa170a668266da0d3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36347 } } peers { permanent_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34661 } }
I20240503 08:15:54.015475 18235 ts_tablet_manager.cc:1398] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:15:54.015475 18236 ts_tablet_manager.cc:1398] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:15:54.015739 17940 heartbeater.cc:503] Master 127.6.68.254:45275 was elected leader, sending a full tablet report...
I20240503 08:15:54.015946 18213 heartbeater.cc:503] Master 127.6.68.254:45275 was elected leader, sending a full tablet report...
May 03 08:15:54 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
W20240503 08:15:54.194192 18214 tablet.cc:2369] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:15:54.237942 18079 tablet.cc:2369] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:15:54.251727 17941 tablet.cc:2369] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:15:54.370157 18234 raft_consensus.cc:482] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:15:54.370363 18234 raft_consensus.cc:504] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b439da89a2bf4ad4870914acb7c79497" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33051 } } peers { permanent_uuid: "0227d75f2082486fa170a668266da0d3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36347 } } peers { permanent_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34661 } }
I20240503 08:15:54.370743 18234 leader_election.cc:288] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b439da89a2bf4ad4870914acb7c79497 (127.6.68.193:33051), 0227d75f2082486fa170a668266da0d3 (127.6.68.195:36347)
I20240503 08:15:54.374624 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:54.371093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:60427 (local address 127.6.68.193:33051)
0503 08:15:54.371140 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:54.371142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:54.371361 (+ 219us) server_negotiation.cc:407] Connection header received
0503 08:15:54.371474 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:54.371475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:54.371485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:54.371514 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:15:54.372016 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.372782 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:54.373892 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.374375 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:54.374385 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:54.374511 (+ 126us) server_negotiation.cc:299] Negotiation successful
0503 08:15:54.374550 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:15:54.374634 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:54.371262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:54451 (local address 127.6.68.195:36347)
0503 08:15:54.371308 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:54.371310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:54.371417 (+ 107us) server_negotiation.cc:407] Connection header received
0503 08:15:54.371521 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:54.371522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:54.371531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:54.371558 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:15:54.372061 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.372777 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:54.373928 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.374382 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:54.374391 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:54.374516 (+ 125us) server_negotiation.cc:299] Negotiation successful
0503 08:15:54.374554 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:15:54.374883 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:54.371138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:36347 (local address 127.6.68.194:54451)
0503 08:15:54.371343 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:54.371354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:54.371444 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:54.371658 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:54.371658 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:54.371660 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:15:54.371950 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:54.371953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.372915 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:54.372916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:54.373810 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:54.373813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.374424 (+ 611us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:54.374427 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:15:54.374751 (+ 324us) client_negotiation.cc:240] Negotiation successful
0503 08:15:54.374793 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":51,"threads_started":1}
I20240503 08:15:54.375221 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:54.370962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:33051 (local address 127.6.68.194:60427)
0503 08:15:54.371274 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:54.371285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:15:54.371392 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:15:54.371596 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:15:54.371597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:15:54.371599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:15:54.371904 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:54.371908 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.372891 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:15:54.372891 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:15:54.373787 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:15:54.373789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.374418 (+ 629us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:54.374422 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:15:54.375098 (+ 676us) client_negotiation.cc:240] Negotiation successful
0503 08:15:54.375142 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":63,"threads_started":1}
I20240503 08:15:54.375209 18167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2f6405dcf4aa89ae79334b4346467" candidate_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0227d75f2082486fa170a668266da0d3" is_pre_election: true
I20240503 08:15:54.375358 18167 raft_consensus.cc:2454] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25dfd08db4a4e99b6c3c261f76015dc in term 0.
I20240503 08:15:54.375581 17965 leader_election.cc:302] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0227d75f2082486fa170a668266da0d3, c25dfd08db4a4e99b6c3c261f76015dc; no voters:
I20240503 08:15:54.375797 18234 raft_consensus.cc:2781] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:15:54.375880 18234 raft_consensus.cc:482] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:15:54.375912 18234 raft_consensus.cc:3037] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:54.376708 18234 raft_consensus.cc:504] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b439da89a2bf4ad4870914acb7c79497" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33051 } } peers { permanent_uuid: "0227d75f2082486fa170a668266da0d3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36347 } } peers { permanent_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34661 } }
I20240503 08:15:54.376850 18234 leader_election.cc:288] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [CANDIDATE]: Term 1 election: Requested vote from peers b439da89a2bf4ad4870914acb7c79497 (127.6.68.193:33051), 0227d75f2082486fa170a668266da0d3 (127.6.68.195:36347)
I20240503 08:15:54.377050 17877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2f6405dcf4aa89ae79334b4346467" candidate_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b439da89a2bf4ad4870914acb7c79497" is_pre_election: true
I20240503 08:15:54.377184 17877 raft_consensus.cc:2454] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25dfd08db4a4e99b6c3c261f76015dc in term 0.
I20240503 08:15:54.377537 17877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2f6405dcf4aa89ae79334b4346467" candidate_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b439da89a2bf4ad4870914acb7c79497"
I20240503 08:15:54.377609 17877 raft_consensus.cc:3037] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:54.378445 18167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2f6405dcf4aa89ae79334b4346467" candidate_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0227d75f2082486fa170a668266da0d3"
I20240503 08:15:54.378484 17877 raft_consensus.cc:2454] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25dfd08db4a4e99b6c3c261f76015dc in term 1.
I20240503 08:15:54.378536 18167 raft_consensus.cc:3037] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:15:54.378643 17965 leader_election.cc:302] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b439da89a2bf4ad4870914acb7c79497, c25dfd08db4a4e99b6c3c261f76015dc; no voters:
I20240503 08:15:54.378736 18234 raft_consensus.cc:2781] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:15:54.378885 18234 raft_consensus.cc:686] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 1 LEADER]: Becoming Leader. State: Replica: c25dfd08db4a4e99b6c3c261f76015dc, State: Running, Role: LEADER
I20240503 08:15:54.378979 18234 consensus_queue.cc:237] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b439da89a2bf4ad4870914acb7c79497" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33051 } } peers { permanent_uuid: "0227d75f2082486fa170a668266da0d3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36347 } } peers { permanent_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34661 } }
I20240503 08:15:54.379463 18167 raft_consensus.cc:2454] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25dfd08db4a4e99b6c3c261f76015dc in term 1.
I20240503 08:15:54.379984 17727 catalog_manager.cc:5484] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc reported cstate change: term changed from 0 to 1, leader changed from <none> to c25dfd08db4a4e99b6c3c261f76015dc (127.6.68.194). New cstate: current_term: 1 leader_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b439da89a2bf4ad4870914acb7c79497" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0227d75f2082486fa170a668266da0d3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c25dfd08db4a4e99b6c3c261f76015dc" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34661 } health_report { overall_health: HEALTHY } } }
May 03 08:15:54 dist-test-slave-3g67 krb5kdc[17252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724153, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:15:54.442672 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:54.440320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.6.68.194:34661)
0503 08:15:54.440374 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0503 08:15:54.440376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:15:54.440385 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:15:54.440449 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:15:54.440450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:15:54.440466 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:15:54.440534 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:15:54.440805 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.441431 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:15:54.442168 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:15:54.442309 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:15:54.442347 (+ 38us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:15:54.442430 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:15:54.442503 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0503 08:15:54.442563 (+ 60us) server_negotiation.cc:299] Negotiation successful
0503 08:15:54.442586 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240503 08:15:54.444474 18167 raft_consensus.cc:1261] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3 [term 1 FOLLOWER]: Refusing update from remote peer c25dfd08db4a4e99b6c3c261f76015dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:15:54.444474 17877 raft_consensus.cc:1261] T 96a2f6405dcf4aa89ae79334b4346467 P b439da89a2bf4ad4870914acb7c79497 [term 1 FOLLOWER]: Refusing update from remote peer c25dfd08db4a4e99b6c3c261f76015dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:15:54.444828 18234 consensus_queue.cc:1035] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b439da89a2bf4ad4870914acb7c79497" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:54.445029 18244 consensus_queue.cc:1035] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0227d75f2082486fa170a668266da0d3" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:15:54.447603 18249 mvcc.cc:205] Tried to move back new op lower bound from 7023510136602017792 to 7023510136336797696. Current Snapshot: MvccSnapshot[applied={T|T < 7023510136602017792}]
I20240503 08:15:54.449728 17727 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38854:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:15:54.449849 17727 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38854:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:15:54.455804 17727 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ad6b03740b904a238786ee77002eda35: Sending DeleteTablet for 3 replicas of tablet 96a2f6405dcf4aa89ae79334b4346467
I20240503 08:15:54.456370 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 17809
I20240503 08:15:54.456432 17855 tablet_service.cc:1514] Processing DeleteTablet for tablet 96a2f6405dcf4aa89ae79334b4346467 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:15:54 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:44698
I20240503 08:15:54.456863 18135 tablet_service.cc:1514] Processing DeleteTablet for tablet 96a2f6405dcf4aa89ae79334b4346467 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:15:54 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:39762
I20240503 08:15:54.457051 17993 tablet_service.cc:1514] Processing DeleteTablet for tablet 96a2f6405dcf4aa89ae79334b4346467 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:15:54 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:55284
I20240503 08:15:54.459769 18258 tablet_replica.cc:330] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: stopping tablet replica
I20240503 08:15:54.459901 18258 raft_consensus.cc:2229] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:15:54.460023 18258 raft_consensus.cc:2258] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:54.460506 18258 ts_tablet_manager.cc:1875] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:15:54.462546 18258 ts_tablet_manager.cc:1888] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:15:54.462636 18258 log.cc:1196] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/96a2f6405dcf4aa89ae79334b4346467
I20240503 08:15:54.462940 18258 ts_tablet_manager.cc:1909] T 96a2f6405dcf4aa89ae79334b4346467 P c25dfd08db4a4e99b6c3c261f76015dc: Deleting consensus metadata
I20240503 08:15:54.463346 17712 catalog_manager.cc:4830] TS c25dfd08db4a4e99b6c3c261f76015dc (127.6.68.194:34661): tablet 96a2f6405dcf4aa89ae79334b4346467 (table test-table [id=28d5bac7dead4056aa20a4fcfa48d84d]) successfully deleted
W20240503 08:15:54.465274 17713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:33051 (error 108)
I20240503 08:15:54.471788 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 17946
I20240503 08:15:54.472494 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:15:54.471924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:33051 (local address 127.0.0.1:44700)
0503 08:15:54.472411 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0503 08:15:54.472455 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466}
I20240503 08:15:54.473743 18257 tablet_replica.cc:330] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: stopping tablet replica
I20240503 08:15:54.473855 18257 raft_consensus.cc:2229] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:15:54.474006 18257 raft_consensus.cc:2258] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:15:54.474474 18257 ts_tablet_manager.cc:1875] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:15:54.475057 17713 catalog_manager.cc:4573] TS b439da89a2bf4ad4870914acb7c79497 (127.6.68.193:33051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96a2f6405dcf4aa89ae79334b4346467: Network error: Client connection negotiation failed: client connection to 127.6.68.193:33051: connect: Connection refused (error 111)
I20240503 08:15:54.476294 18257 ts_tablet_manager.cc:1888] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:15:54.476410 18257 log.cc:1196] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/96a2f6405dcf4aa89ae79334b4346467
I20240503 08:15:54.476706 18257 ts_tablet_manager.cc:1909] T 96a2f6405dcf4aa89ae79334b4346467 P 0227d75f2082486fa170a668266da0d3: Deleting consensus metadata
I20240503 08:15:54.477144 17713 catalog_manager.cc:4830] TS 0227d75f2082486fa170a668266da0d3 (127.6.68.195:36347): tablet 96a2f6405dcf4aa89ae79334b4346467 (table test-table [id=28d5bac7dead4056aa20a4fcfa48d84d]) successfully deleted
I20240503 08:15:54.481182 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 18084
I20240503 08:15:54.487594 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 17696
I20240503 08:15:54.499848 6419 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20240503 08:15:54.834170 6419 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20240503 08:15:54.834307 6419 mini_ranger.cc:66] Stopping Ranger...
I20240503 08:15:55.181560 6419 mini_ranger.cc:68] Stopped Ranger
2024-05-03 08:15:55.181 UTC [17282] LOG: received smart shutdown request
2024-05-03 08:15:55.185 UTC [17282] LOG: background worker "logical replication launcher" (PID 17291) exited with exit code 1
2024-05-03 08:15:55.185 UTC [17286] LOG: shutting down
2024-05-03 08:15:55.352 UTC [17282] LOG: database system is shut down
2024-05-03T08:15:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44665 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:15:55 dist-test-slave-3g67 krb5kdc[18268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:15:55 dist-test-slave-3g67 krb5kdc[18268](info): set up 2 sockets
May 03 08:15:55 dist-test-slave-3g67 krb5kdc[18268](info): commencing operation
krb5kdc: starting...
W20240503 08:15:56.419901 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:15:56 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:15:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/rangeradmin_127.6.68.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/rangeradmin_127.6.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/rangerlookup_127.6.68.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/rangerlookup_127.6.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/HTTP_127.6.68.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/HTTP_127.6.68.212@KRBTEST.COM.keytab.
I20240503 08:15:56.582628 6419 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxRG6Mz/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/postgres -l logfile start
2024-05-03 08:15:59.196 UTC [18298] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-03 08:15:59.197 UTC [18298] LOG: listening on IPv4 address "127.6.68.212", port 43219
2024-05-03 08:15:59.200 UTC [18298] LOG: listening on Unix socket "/tmp/.s.PGSQL.43219"
2024-05-03 08:15:59.221 UTC [18301] LOG: database system was shut down at 2024-05-03 08:15:57 UTC
2024-05-03 08:15:59.224 UTC [18298] LOG: database system is ready to accept connections
I20240503 08:16:00.087954 6419 mini_postgres.cc:96] Postgres bound to 43219
2024-05-03 08:16:00.093 UTC [18309] FATAL: database "slave" does not exist
127.6.68.212:43219 - accepting connections
I20240503 08:16:00.094252 6419 mini_ranger.cc:161] Starting Ranger...
I20240503 08:16:00.108458 6419 mini_ranger.cc:84] Created miniranger Postgres user
I20240503 08:16:00.632014 6419 mini_ranger.cc:87] Created ranger Postgres database
I20240503 08:16:00.632102 6419 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-admin
2024-05-03 08:16:00,869 [I] DB FLAVOR :POSTGRES
2024-05-03 08:16:00,870 [I] --------- Verifying Ranger DB connection ---------
2024-05-03 08:16:00,870 [I] Checking connection..
2024-05-03 08:16:00,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-03 08:16:01,127 [I] Checking connection passed.
2024-05-03 08:16:01,127 [I] --------- Verifying version history table ---------
2024-05-03 08:16:01,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-03 08:16:01,385 [I] Table x_db_version_h does not exist in database ranger
2024-05-03 08:16:01,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-03 08:16:01,638 [I] Table x_db_version_h does not exist in database ranger
2024-05-03 08:16:01,639 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-03 08:16:01,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-03 08:16:01.866 UTC [18373] WARNING: there is no transaction in progress
2024-05-03 08:16:01,873 [I] create_dbversion_catalog.sql file imported successfully
2024-05-03 08:16:01,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-03 08:16:02,121 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-03 08:16:02,121 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-03 08:16:02,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-03 08:16:02,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-03 08:16:02,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3g67.c.gcp-upstream.internal','N') ;"
2024-05-03 08:16:02,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-03 08:16:03,101 [I] Table x_portal_user does not exist in database ranger
2024-05-03 08:16:03,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-03 08:16:03,340 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-03 08:16:03,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-03 08:16:03,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-03 08:16:03,834 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-03 08:16:03,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-03 08:16:04.797 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:04.813 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:04.830 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:04.845 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:04.862 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:04.882 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:04.946 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:04.954 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:04.968 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:04.978 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:04.988 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:05.002 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:05.012 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:05.018 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:05.025 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:05.032 UTC [18481] WARNING: there is no transaction in progress
2024-05-03 08:16:05,710 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-03 08:16:05,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-03 08:16:05,953 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-03 08:16:05,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-03 08:16:06,205 [I] Table x_portal_user already exists in database 'ranger'
2024-05-03 08:16:06,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-03 08:16:06,441 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-03 08:16:06,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-03 08:16:06,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-03 08:16:06,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3g67.c.gcp-upstream.internal';"
2024-05-03 08:16:07,157 [I] CORE_DB_SCHEMA import status has been updated
2024-05-03 08:16:07,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-03 08:16:07,382 [I] DB_PATCHES have already been applied
I20240503 08:16:07.388288 6419 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxRG6Mz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240503 08:16:07.388371 6419 mini_ranger.cc:193] Using host: 127.6.68.212
I20240503 08:16:07.391059 6419 mini_ranger.cc:227] Ranger admin URL: http://127.6.68.212:60817
May 03, 2024 8:16:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 03, 2024 8:16:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 03, 2024 8:16:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-admin/ews/webapp
May 03, 2024 8:16:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 03, 2024 8:16:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-admin/ews/webapp] .....
May 03, 2024 8:16:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-admin/ews/webapp].
May 03, 2024 8:16:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20240503 08:16:08.428673 6419 mini_ranger.cc:160] Time spent starting Ranger: real 8.334s user 0.001s sys 0.008s
May 03, 2024 8:16:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 03, 2024 8:16:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
May 03, 2024 8:16:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 03, 2024 8:16:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 03, 2024 8:16:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240503 08:16:28.414170 6419 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.68.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.68.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/rangerkms_127.6.68.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.68.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/rangerkms_127.6.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.68.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.68.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.68.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/HTTP_127.6.68.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.68.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/HTTP_127.6.68.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 03 08:16:28 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724188, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20240503 08:16:28.486526 6419 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20240503 08:16:28.494362 6419 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20240503 08:16:29.403717 6419 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20240503 08:16:29.403784 6419 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-kms
I20240503 08:16:29.403796 6419 mini_ranger_kms.cc:227] Using postgres at 127.6.68.212:43219
2024-05-03 08:16:29,561 [I] DB FLAVOR :POSTGRES
2024-05-03 08:16:29,561 [I] --------- Verifying Ranger DB connection ---------
2024-05-03 08:16:29,561 [I] Checking connection
2024-05-03 08:16:29,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2024-05-03 08:16:29,792 [I] connection success
2024-05-03 08:16:29,792 [I] --------- Verifying Ranger DB tables ---------
2024-05-03 08:16:29,792 [I] Verifying table ranger_masterkey in database rangerkms
2024-05-03 08:16:29,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2024-05-03 08:16:30,023 [I] Table ranger_masterkey does not exist in database rangerkms
2024-05-03 08:16:30,023 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-03 08:16:30,023 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2024-05-03 08:16:30,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.68.212:43219/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2024-05-03 08:16:30,299 [I] kms_core_db_postgres.sql DB schema imported successfully
I20240503 08:16:30.747385 6419 mini_ranger_kms.cc:310] Created kms service
I20240503 08:16:30.966753 6419 mini_ranger_kms.cc:326] Created kudu user
I20240503 08:16:31.006959 6419 mini_ranger_kms.cc:343] Created rangerkms user
May 03 08:16:31 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724188, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:16:31.390952 6419 mini_ranger_kms.cc:384] Added ranger policy
I20240503 08:16:31.391131 6419 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxRG6Mz/build/release/bin/postgresql.jar:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxRG6Mz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20240503 08:16:31.391153 6419 mini_ranger_kms.cc:240] Using host: 127.6.68.212
I20240503 08:16:31.393909 6419 mini_ranger_kms.cc:276] Ranger KMS PID: 18661
I20240503 08:16:31.393985 6419 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.68.212:58321
May 03, 2024 8:16:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 03, 2024 8:16:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 03, 2024 8:16:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 03, 2024 8:16:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-kms/ews/webapp] .....
May 03, 2024 8:16:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-kms/ews/webapp].
May 03, 2024 8:16:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20240503 08:16:33.390269 6419 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.904s user 0.002s sys 0.139s
I20240503 08:16:33.390377 6419 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20240503 08:16:33.390425 6419 mini_ranger_kms.cc:401] 127.6.68.212:58321/kms/v1/keys
May 03 08:16:33 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714724193, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.68.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 03 08:16:33 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724193, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.68.212@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
May 03 08:16:34 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724193, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.68.212@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
May 03 08:16:36 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:16:36.343345 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34417
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:34417
--ranger_config_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:58321/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:36.422946 18712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:36.423156 18712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:36.423177 18712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:36.424520 18712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:16:36.424564 18712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:36.424577 18712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:36.424589 18712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:16:36.424600 18712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:16:36.426172 18712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.68.212:58321/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:34417
--ranger_config_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:36.426385 18712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:36.426587 18712 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:36.431324 18718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:36.431979 18717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:36.432538 18720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:36.433143 18712 server_base.cc:1034] running on GCE node
I20240503 08:16:36.433372 18712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:36.433612 18712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:36.434752 18712 hybrid_clock.cc:648] HybridClock initialized: now 1714724196434735 us; error 40 us; skew 500 ppm
May 03 08:16:36 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724196, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:36.447474 18712 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:16:36.447984 18712 webserver.cc:460] Webserver started at http://127.6.68.254:39919/ using document root <none> and password file <none>
I20240503 08:16:36.448175 18712 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:36.448216 18712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:36.448338 18712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 03 08:16:36 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724196, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:16:36.494138 18712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "b7528708cd11492aa571a3057ede9400"
format_stamp: "Formatted at 2024-05-03 08:16:36 on dist-test-slave-3g67"
server_key: "19bc5f335aba3267b656ef98cc8f6f6d"
server_key_iv: "fc6ad9c4361dad775a3d9bce9920bbd0"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:36.494494 18712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "b7528708cd11492aa571a3057ede9400"
format_stamp: "Formatted at 2024-05-03 08:16:36 on dist-test-slave-3g67"
server_key: "19bc5f335aba3267b656ef98cc8f6f6d"
server_key_iv: "fc6ad9c4361dad775a3d9bce9920bbd0"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:36.495782 18712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20240503 08:16:36.507539 18727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:36.507813 18712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:36.507906 18712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "b7528708cd11492aa571a3057ede9400"
format_stamp: "Formatted at 2024-05-03 08:16:36 on dist-test-slave-3g67"
server_key: "19bc5f335aba3267b656ef98cc8f6f6d"
server_key_iv: "fc6ad9c4361dad775a3d9bce9920bbd0"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:36.507994 18712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:36.529470 18712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:36.531438 18712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:36.531543 18712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:36.536140 18712 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:34417
I20240503 08:16:36.536180 18779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:34417 every 8 connection(s)
I20240503 08:16:36.536489 18712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:16:36.537189 18780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:36.538756 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 18712
I20240503 08:16:36.538875 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:16:36.539034 6419 external_mini_cluster.cc:1428] Setting key 339675197090184d9c7cc5b2e6a54547
I20240503 08:16:36.540210 18780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400: Bootstrap starting.
I20240503 08:16:36.540904 18780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:36.541182 18780 log.cc:824] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:36.541914 18780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400: No bootstrap required, opened a new log
I20240503 08:16:36.544077 18780 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } }
I20240503 08:16:36.544239 18780 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:36.544276 18780 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7528708cd11492aa571a3057ede9400, State: Initialized, Role: FOLLOWER
I20240503 08:16:36.544428 18780 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } }
I20240503 08:16:36.544514 18780 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:16:36.544586 18780 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:16:36.544674 18780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:36.545476 18780 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } }
I20240503 08:16:36.545580 18780 leader_election.cc:302] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7528708cd11492aa571a3057ede9400; no voters:
I20240503 08:16:36.545768 18780 leader_election.cc:288] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:16:36.545820 18785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 1 FOLLOWER]: Leader election won for term 1
May 03 08:16:36 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:36.546008 18785 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 1 LEADER]: Becoming Leader. State: Replica: b7528708cd11492aa571a3057ede9400, State: Running, Role: LEADER
I20240503 08:16:36.546097 18785 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } }
I20240503 08:16:36.546141 18780 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:16:36.546408 18785 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7528708cd11492aa571a3057ede9400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } } }
I20240503 08:16:36.546468 18785 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:36.546586 18785 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7528708cd11492aa571a3057ede9400. Latest consensus state: current_term: 1 leader_uuid: "b7528708cd11492aa571a3057ede9400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } } }
I20240503 08:16:36.546671 18785 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:36.547653 18789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:16:36.548000 18780 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240503 08:16:36.549346 18789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:16:36.550343 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:36.539580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.6.68.254:34417)
0503 08:16:36.539776 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:36.539788 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:36.539815 (+ 27us) server_negotiation.cc:407] Connection header received
0503 08:16:36.540302 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:36.540310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:36.540368 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:36.540508 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:36.541456 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:36.542619 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:36.543431 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:36.543701 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:36.546564 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:36.546574 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:36.546574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:36.546594 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:36.548802 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:36.549240 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:36.549243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:36.549243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:36.549298 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:36.549494 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:36.549495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:36.549495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:36.549712 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:36.549876 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:36.550008 (+ 132us) server_negotiation.cc:299] Negotiation successful
0503 08:16:36.550116 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20240503 08:16:36.555065 18789 catalog_manager.cc:1347] Generated new cluster ID: 67162997f46b48c1992d62f4e0ed760f
I20240503 08:16:36.555164 18789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:16:36.571731 18789 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:16:36.572713 18789 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:16:36.579593 18789 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400: Generated new TSK 0
I20240503 08:16:36.579926 18789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:16:37.360551 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:58321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34417
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:37.450903 18825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:37.451112 18825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:37.451138 18825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:37.452597 18825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:37.452654 18825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:37.452682 18825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:16:37.454231 18825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.68.212:58321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:37.454442 18825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:37.454644 18825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:37.459494 18830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:37.460000 18833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:37.460772 18831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:37.462568 18825 server_base.cc:1034] running on GCE node
I20240503 08:16:37.462807 18825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:37.463059 18825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:37.464205 18825 hybrid_clock.cc:648] HybridClock initialized: now 1714724197464193 us; error 36 us; skew 500 ppm
May 03 08:16:37 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:37.466778 18825 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:16:37.467185 18825 webserver.cc:460] Webserver started at http://127.6.68.193:39315/ using document root <none> and password file <none>
I20240503 08:16:37.467376 18825 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:37.467424 18825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:37.467533 18825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 03 08:16:37 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:16:37.482674 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "4987d2a6c60e42e48c48b947361b8e17"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "d008154b19edb55a1c0ea60c2cabccdd"
server_key_iv: "142f717d5033be163d68f365e55e354c"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:37.483285 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "4987d2a6c60e42e48c48b947361b8e17"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "d008154b19edb55a1c0ea60c2cabccdd"
server_key_iv: "142f717d5033be163d68f365e55e354c"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:37.485013 18825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:37.503206 18840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:37.503540 18825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:37.503671 18825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "4987d2a6c60e42e48c48b947361b8e17"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "d008154b19edb55a1c0ea60c2cabccdd"
server_key_iv: "142f717d5033be163d68f365e55e354c"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:37.503769 18825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:37.524065 18825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:37.526568 18825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:37.526777 18825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:37.527043 18825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:37.527446 18825 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:37.527498 18825 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:37.527530 18825 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:37.527581 18825 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:37.534583 18825 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:36615
I20240503 08:16:37.534721 18955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:36615 every 8 connection(s)
I20240503 08:16:37.535007 18825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:16:37.535851 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 18825
I20240503 08:16:37.535957 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:16:37.536154 6419 external_mini_cluster.cc:1428] Setting key fa223f6133c79f7036248c260681e6f7
May 03 08:16:37 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:37.544977 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:37.536083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:47019 (local address 127.6.68.254:34417)
0503 08:16:37.536237 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:37.536240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:37.536256 (+ 16us) server_negotiation.cc:407] Connection header received
0503 08:16:37.537193 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:37.537194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:37.537208 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:37.537290 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:37.538138 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:37.538645 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:37.539234 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:37.539404 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:37.541523 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:37.541528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:37.541529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:37.541536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:37.543693 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:37.544093 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:37.544094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:37.544094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:37.544125 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:37.544310 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:37.544311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:37.544311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:37.544421 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:37.544484 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:37.544757 (+ 273us) server_negotiation.cc:299] Negotiation successful
0503 08:16:37.544865 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20240503 08:16:37.545069 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:37.535766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34417 (local address 127.6.68.193:47019)
0503 08:16:37.535966 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:37.536004 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:37.537066 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:37.537349 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:37.537354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:37.537616 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:37.538043 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:37.538049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:37.538749 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:37.538750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:37.539156 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:37.539158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:37.539295 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:37.539916 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:37.539932 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:37.541411 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:37.543779 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:37.543794 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:37.543798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:37.544018 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:37.544184 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:37.544185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:37.544186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:37.544250 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:37.544480 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:37.544486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:37.544617 (+ 131us) client_negotiation.cc:769] Sending connection context
0503 08:16:37.544731 (+ 114us) client_negotiation.cc:240] Negotiation successful
0503 08:16:37.544870 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20240503 08:16:37.546074 18956 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34417
I20240503 08:16:37.546180 18956 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
I20240503 08:16:37.546448 18956 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
I20240503 08:16:37.547058 18744 ts_manager.cc:194] Registered new tserver with Master: 4987d2a6c60e42e48c48b947361b8e17 (127.6.68.193:36615)
I20240503 08:16:37.547888 18744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:47019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:16:37.570988 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:58321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34417
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:37.679235 18962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:37.679512 18962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:37.679548 18962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:37.681710 18962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:37.681799 18962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:37.681840 18962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:16:37.684319 18962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.68.212:58321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:37.684587 18962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:37.684823 18962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:37.690678 18969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:37.691128 18970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:37.691833 18972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:37.692937 18962 server_base.cc:1034] running on GCE node
I20240503 08:16:37.693176 18962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:37.693454 18962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:37.694619 18962 hybrid_clock.cc:648] HybridClock initialized: now 1714724197694593 us; error 45 us; skew 500 ppm
May 03 08:16:37 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:37.697837 18962 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:16:37.698374 18962 webserver.cc:460] Webserver started at http://127.6.68.194:45345/ using document root <none> and password file <none>
I20240503 08:16:37.698590 18962 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:37.698660 18962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:37.698781 18962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 03 08:16:37 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
May 03 08:16:37 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714724197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:37.717576 18962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "1407a39aef8c6f68a74afee4450bdc7f"
server_key_iv: "35956ea5692de6a3e8350eda657552d6"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:37.718003 18962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "1407a39aef8c6f68a74afee4450bdc7f"
server_key_iv: "35956ea5692de6a3e8350eda657552d6"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:37.719764 18962 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:16:37.746057 18979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:37.747143 18962 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20240503 08:16:37.747270 18962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "1407a39aef8c6f68a74afee4450bdc7f"
server_key_iv: "35956ea5692de6a3e8350eda657552d6"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:37.747372 18962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:37.761274 18962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:37.763676 18962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:37.763818 18962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:37.764022 18962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:37.764380 18962 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:37.764425 18962 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:37.764461 18962 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:37.764482 18962 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:37.770807 18962 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:41835
I20240503 08:16:37.771234 18962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:16:37.773857 19092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:41835 every 8 connection(s)
I20240503 08:16:37.777654 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 18962
I20240503 08:16:37.777765 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:16:37.777931 6419 external_mini_cluster.cc:1428] Setting key 3e2d89b0c5a645428d60d4ce6f21f655
May 03 08:16:37 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:37.787274 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:37.774710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:36919 (local address 127.6.68.254:34417)
0503 08:16:37.774764 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:37.774766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:37.775055 (+ 289us) server_negotiation.cc:407] Connection header received
0503 08:16:37.776245 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:37.776246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:37.776264 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:37.776295 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:37.777332 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:37.778161 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:37.781158 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:37.781355 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:37.783936 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:37.783945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:37.783945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:37.783953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:37.785745 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:37.786259 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:37.786261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:37.786261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:37.786290 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:37.786507 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:37.786508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:37.786509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:37.786627 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:37.786668 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:37.787045 (+ 377us) server_negotiation.cc:299] Negotiation successful
0503 08:16:37.787148 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240503 08:16:37.787792 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:37.774451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34417 (local address 127.6.68.194:36919)
0503 08:16:37.774950 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:37.774980 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:37.776090 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:37.776485 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:37.776490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:37.776767 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:37.777210 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:37.777215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:37.780406 (+ 3191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:37.780407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:37.781050 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:37.781053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:37.781502 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:37.782160 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:37.782187 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:37.783814 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:37.785853 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:37.785859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:37.785863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:37.786156 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:37.786358 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:37.786359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:37.786359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:37.786445 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:37.786710 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:37.786715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:37.786876 (+ 161us) client_negotiation.cc:769] Sending connection context
0503 08:16:37.787426 (+ 550us) client_negotiation.cc:240] Negotiation successful
0503 08:16:37.787566 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":83,"threads_started":1}
I20240503 08:16:37.788236 19093 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34417
I20240503 08:16:37.788337 19093 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:37.788578 19093 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
I20240503 08:16:37.788931 18744 ts_manager.cc:194] Registered new tserver with Master: 8cf6aeccea5b453ab3e97b15fbdfee9e (127.6.68.194:41835)
I20240503 08:16:37.789469 18744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:36919
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:16:37.827119 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:58321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34417
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:37.953984 19099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:37.954320 19099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:37.954408 19099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:37.956744 19099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:37.956894 19099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:37.956981 19099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:16:37.959534 19099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.68.212:58321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:37.960045 19099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:37.960311 19099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:37.966708 19107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:37.967355 19104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:37.968088 19105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:37.968761 19099 server_base.cc:1034] running on GCE node
I20240503 08:16:37.969008 19099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:37.969321 19099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:37.971217 19099 hybrid_clock.cc:648] HybridClock initialized: now 1714724197971199 us; error 33 us; skew 500 ppm
May 03 08:16:37 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:37.974551 19099 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:16:37.975112 19099 webserver.cc:460] Webserver started at http://127.6.68.195:33149/ using document root <none> and password file <none>
I20240503 08:16:37.975322 19099 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:37.975365 19099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:37.975471 19099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 03 08:16:37 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:16:37.995254 19099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "224927359bb74d81822f433620121a04"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "3ecf6513037ad77c52a4eed83e1c22da"
server_key_iv: "8637f71f5b4c2e8cee8b69a1d4457a49"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:37.998984 19099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "224927359bb74d81822f433620121a04"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "3ecf6513037ad77c52a4eed83e1c22da"
server_key_iv: "8637f71f5b4c2e8cee8b69a1d4457a49"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:38.002264 19099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240503 08:16:38.016711 19114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.016964 19099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:38.017069 19099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "224927359bb74d81822f433620121a04"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "3ecf6513037ad77c52a4eed83e1c22da"
server_key_iv: "8637f71f5b4c2e8cee8b69a1d4457a49"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:38.017163 19099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:38.031885 19099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:38.033919 19099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:38.034055 19099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:38.034287 19099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:38.034629 19099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:38.034668 19099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.034703 19099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:38.034724 19099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.050302 19099 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:43161
I20240503 08:16:38.050853 19099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:16:38.053305 19227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:43161 every 8 connection(s)
I20240503 08:16:38.058776 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 19099
I20240503 08:16:38.058897 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:16:38.059091 6419 external_mini_cluster.cc:1428] Setting key 14e54f392950fd56788ec4f2143608f0
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724197, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:38.065186 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:38.054202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:42877 (local address 127.6.68.254:34417)
0503 08:16:38.054258 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:38.054260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:38.055081 (+ 821us) server_negotiation.cc:407] Connection header received
0503 08:16:38.055778 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:38.055779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:38.055794 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:38.055824 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:38.056814 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.057632 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:38.058427 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.058594 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:38.061468 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:38.061472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:38.061472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:38.061479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:38.063472 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:38.064232 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:38.064233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:38.064234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:38.064261 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:38.064469 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:38.064469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:38.064470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:38.064590 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:38.064639 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:38.064978 (+ 339us) server_negotiation.cc:299] Negotiation successful
0503 08:16:38.065081 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240503 08:16:38.065656 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:38.054531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34417 (local address 127.6.68.195:42877)
0503 08:16:38.054961 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:38.054993 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:38.055646 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:38.056007 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:38.056012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:38.056297 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:38.056703 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:38.056708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.057750 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:38.057751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:38.058329 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:38.058331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.058724 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:38.059404 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:38.059427 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:38.061371 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:38.063816 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:38.063824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:38.063828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:38.064130 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:38.064328 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:38.064328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:38.064329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:38.064408 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:38.064666 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:38.064671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:38.064823 (+ 152us) client_negotiation.cc:769] Sending connection context
0503 08:16:38.065316 (+ 493us) client_negotiation.cc:240] Negotiation successful
0503 08:16:38.065452 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":132,"threads_started":1}
I20240503 08:16:38.066200 19228 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34417
I20240503 08:16:38.066313 19228 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:38.066624 19228 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
I20240503 08:16:38.067024 18744 ts_manager.cc:194] Registered new tserver with Master: 224927359bb74d81822f433620121a04 (127.6.68.195:43161)
I20240503 08:16:38.067564 18744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:42877
I20240503 08:16:38.070901 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:16:38.071455 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 18825
I20240503 08:16:38.081254 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 18962
I20240503 08:16:38.087446 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 19099
I20240503 08:16:38.093299 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 18712
I20240503 08:16:38.100957 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34417
--webserver_interface=127.6.68.254
--webserver_port=39919
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:34417
--ranger_config_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:58321/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.68.212:58321/kms with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:38.200310 19231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:38.200608 19231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:38.200662 19231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:38.202500 19231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:16:38.202567 19231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:38.202589 19231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:38.202610 19231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:16:38.202630 19231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:16:38.204420 19231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.68.212:58321/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:34417
--ranger_config_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=39919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:38.204670 19231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:38.204854 19231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:38.210637 19236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:38.211205 19239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:38.212994 19237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:38.213580 19231 server_base.cc:1034] running on GCE node
I20240503 08:16:38.213829 19231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:38.214084 19231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:38.215250 19231 hybrid_clock.cc:648] HybridClock initialized: now 1714724198215228 us; error 42 us; skew 500 ppm
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724198, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:38.217957 19231 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:16:38.218439 19231 webserver.cc:460] Webserver started at http://127.6.68.254:39919/ using document root <none> and password file <none>
I20240503 08:16:38.218642 19231 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:38.218700 19231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724198, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:16:38.263268 19231 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:38.264245 19247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.264520 19231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:16:38.264614 19231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "b7528708cd11492aa571a3057ede9400"
format_stamp: "Formatted at 2024-05-03 08:16:36 on dist-test-slave-3g67"
server_key: "19bc5f335aba3267b656ef98cc8f6f6d"
server_key_iv: "fc6ad9c4361dad775a3d9bce9920bbd0"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:38.264993 19231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:38.282588 19231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:38.285140 19231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:38.285279 19231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:38.290498 19231 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:34417
I20240503 08:16:38.290553 19299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:34417 every 8 connection(s)
I20240503 08:16:38.290937 19231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:16:38.292229 19300 sys_catalog.cc:263] Verifying existing consensus state
I20240503 08:16:38.292841 19300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400: Bootstrap starting.
I20240503 08:16:38.298198 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 19231
I20240503 08:16:38.298641 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:58321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:36615
--local_ip_for_outbound_sockets=127.6.68.193
--tserver_master_addrs=127.6.68.254:34417
--webserver_port=39315
--webserver_interface=127.6.68.193
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.68.212:58321/kms with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:16:38.303246 19300 log.cc:824] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:38.305642 19300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:16:38.305999 19300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400: Bootstrap complete.
I20240503 08:16:38.308413 19300 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } }
I20240503 08:16:38.308629 19300 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7528708cd11492aa571a3057ede9400, State: Initialized, Role: FOLLOWER
I20240503 08:16:38.308779 19300 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } }
I20240503 08:16:38.308845 19300 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:16:38.308887 19300 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:16:38.308925 19300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:16:38.310076 19300 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } }
I20240503 08:16:38.310194 19300 leader_election.cc:302] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7528708cd11492aa571a3057ede9400; no voters:
I20240503 08:16:38.310392 19300 leader_election.cc:288] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 08:16:38.310520 19304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:16:38.310647 19300 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:16:38.310683 19304 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [term 2 LEADER]: Becoming Leader. State: Replica: b7528708cd11492aa571a3057ede9400, State: Running, Role: LEADER
I20240503 08:16:38.310873 19304 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } }
I20240503 08:16:38.311197 19304 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b7528708cd11492aa571a3057ede9400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } } }
I20240503 08:16:38.311355 19304 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:38.311517 19304 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7528708cd11492aa571a3057ede9400. Latest consensus state: current_term: 2 leader_uuid: "b7528708cd11492aa571a3057ede9400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7528708cd11492aa571a3057ede9400" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34417 } } }
I20240503 08:16:38.311738 19304 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:38.311920 19313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:16:38.312233 19300 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240503 08:16:38.312587 19313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:16:38.314370 19313 catalog_manager.cc:1259] Loaded cluster ID: 67162997f46b48c1992d62f4e0ed760f
I20240503 08:16:38.314492 19313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:16:38.317541 19313 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:16:38.317903 19313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400: Loaded TSK: 0
I20240503 08:16:38.318146 19313 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240503 08:16:38.422933 19302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:38.423207 19302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:38.423245 19302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:38.425505 19302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:38.425724 19302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:38.425845 19302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:16:38.428452 19302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.68.212:58321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:36615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=39315
--tserver_master_addrs=127.6.68.254:34417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:38.428989 19302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:38.429296 19302 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:38.435351 19338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:38.435842 19340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:38.437309 19302 server_base.cc:1034] running on GCE node
W20240503 08:16:38.437603 19337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:38.437891 19302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:38.438211 19302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:38.439426 19302 hybrid_clock.cc:648] HybridClock initialized: now 1714724198439413 us; error 30 us; skew 500 ppm
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724198, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:38.442816 19302 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:16:38.443290 19302 webserver.cc:460] Webserver started at http://127.6.68.193:39315/ using document root <none> and password file <none>
I20240503 08:16:38.443488 19302 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:38.443543 19302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724198, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:16:38.461716 19302 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20240503 08:16:38.464288 19348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.464537 19302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20240503 08:16:38.464612 19302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "4987d2a6c60e42e48c48b947361b8e17"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "d008154b19edb55a1c0ea60c2cabccdd"
server_key_iv: "142f717d5033be163d68f365e55e354c"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:38.465018 19302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:38.481889 19302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:38.484742 19302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:38.484958 19302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:38.485249 19302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:38.485648 19302 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:38.485730 19302 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.485844 19302 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:38.485924 19302 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.493589 19302 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:36615
I20240503 08:16:38.494123 19302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:16:38.496522 19461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:36615 every 8 connection(s)
I20240503 08:16:38.498064 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 19302
I20240503 08:16:38.498386 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:58321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:41835
--local_ip_for_outbound_sockets=127.6.68.194
--tserver_master_addrs=127.6.68.254:34417
--webserver_port=45345
--webserver_interface=127.6.68.194
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.68.212:58321/kms with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724198, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:38.525969 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:38.496204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34417 (local address 127.6.68.193:32931)
0503 08:16:38.497096 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:38.497127 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:38.500508 (+ 3381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:38.508350 (+ 7842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:38.508358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:38.508669 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:38.509118 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:38.509125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.510884 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:38.510885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:38.511535 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:38.511539 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.511682 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:38.512367 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:38.512390 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:38.514038 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:38.524440 (+ 10402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:38.524451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:38.524453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:38.524754 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:38.524979 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:38.524980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:38.524980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:38.525050 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:38.525423 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:38.525427 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:38.525552 (+ 125us) client_negotiation.cc:769] Sending connection context
0503 08:16:38.525636 (+ 84us) client_negotiation.cc:240] Negotiation successful
0503 08:16:38.525770 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":87,"threads_started":1}
I20240503 08:16:38.526804 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:38.496279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:32931 (local address 127.6.68.254:34417)
0503 08:16:38.507754 (+ 11475us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:38.507771 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:38.507808 (+ 37us) server_negotiation.cc:407] Connection header received
0503 08:16:38.507972 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:38.507980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:38.508044 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:38.509287 (+ 1243us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:38.509444 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.510694 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:38.512020 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.513036 (+ 1016us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:38.514161 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:38.514192 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:38.514192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:38.514211 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:38.524265 (+ 10054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:38.524852 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:38.524854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:38.524855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:38.524911 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:38.525112 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:38.525113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:38.525113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:38.525347 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:38.526282 (+ 935us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:38.526369 (+ 87us) server_negotiation.cc:299] Negotiation successful
0503 08:16:38.526477 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11412,"thread_start_us":100,"threads_started":1}
I20240503 08:16:38.527531 19462 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34417
I20240503 08:16:38.527647 19462 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:38.527886 19462 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
W20240503 08:16:38.528708 19462 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:34417 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240503 08:16:38.528788 19462 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:38.528880 19462 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
I20240503 08:16:38.529294 19462 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:38.529398 19462 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
W20240503 08:16:38.529944 19462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240503 08:16:38.608582 19466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:38.608852 19466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:38.608894 19466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:38.611135 19466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:38.611249 19466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:38.611330 19466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:16:38.613911 19466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.68.212:58321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:41835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=45345
--tserver_master_addrs=127.6.68.254:34417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:38.614331 19466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:38.614619 19466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:38.620621 19471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:38.621912 19472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:38.622699 19474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:38.623445 19466 server_base.cc:1034] running on GCE node
I20240503 08:16:38.623729 19466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:38.624033 19466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:38.625274 19466 hybrid_clock.cc:648] HybridClock initialized: now 1714724198625253 us; error 36 us; skew 500 ppm
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724198, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:38.627915 19466 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:16:38.628414 19466 webserver.cc:460] Webserver started at http://127.6.68.194:45345/ using document root <none> and password file <none>
I20240503 08:16:38.628618 19466 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:38.628674 19466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724198, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:16:38.646193 19466 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20240503 08:16:38.647724 19482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.647938 19466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:38.648010 19466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "1407a39aef8c6f68a74afee4450bdc7f"
server_key_iv: "35956ea5692de6a3e8350eda657552d6"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:38.648406 19466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:38.659925 19466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:38.662465 19466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:38.662648 19466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:38.662865 19466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:38.663197 19466 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:38.663249 19466 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.663280 19466 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:38.663340 19466 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.670023 19466 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:41835
I20240503 08:16:38.670090 19595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:41835 every 8 connection(s)
I20240503 08:16:38.670514 19466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:16:38.679179 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 19466
I20240503 08:16:38.679558 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.68.212:58321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:43161
--local_ip_for_outbound_sockets=127.6.68.195
--tserver_master_addrs=127.6.68.254:34417
--webserver_port=33149
--webserver_interface=127.6.68.195
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.68.212:58321/kms with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724198, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:38.688853 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:38.671240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:35211 (local address 127.6.68.254:34417)
0503 08:16:38.671290 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:38.671293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:38.671955 (+ 662us) server_negotiation.cc:407] Connection header received
0503 08:16:38.673130 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:38.673131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:38.673143 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:38.673213 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:38.673991 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.674769 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:38.675604 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.675845 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:38.683030 (+ 7185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:38.683037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:38.683038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:38.683047 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:38.684972 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:38.687740 (+ 2768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:38.687742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:38.687742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:38.687785 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:38.688066 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:38.688066 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:38.688067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:38.688193 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:38.688249 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:38.688631 (+ 382us) server_negotiation.cc:299] Negotiation successful
0503 08:16:38.688744 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:16:38.689345 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:38.671373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34417 (local address 127.6.68.194:35211)
0503 08:16:38.671688 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:38.671719 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:38.672987 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:38.673286 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:38.673291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:38.673569 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:38.673885 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:38.673889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.674884 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:38.674885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:38.675507 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:38.675510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.675687 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:38.676412 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:38.676438 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:38.682867 (+ 6429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:38.685101 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:38.685112 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:38.685116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:38.685442 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:38.687870 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:38.687871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:38.687872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:38.687998 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:38.688284 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:38.688289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:38.688455 (+ 166us) client_negotiation.cc:769] Sending connection context
0503 08:16:38.688970 (+ 515us) client_negotiation.cc:240] Negotiation successful
0503 08:16:38.689140 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20240503 08:16:38.689988 19596 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34417
I20240503 08:16:38.690088 19596 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:38.690315 19596 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
W20240503 08:16:38.691010 19596 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:34417 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240503 08:16:38.691085 19596 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:38.691179 19596 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
I20240503 08:16:38.691542 19596 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:38.691684 19596 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
W20240503 08:16:38.692040 19596 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240503 08:16:38.806727 19599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:38.807180 19599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:38.807232 19599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:38.809542 19599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:38.809618 19599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:38.809662 19599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:16:38.812356 19599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.68.212:58321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:43161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=33149
--tserver_master_addrs=127.6.68.254:34417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:38.812625 19599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:38.812886 19599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:38.818548 19604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:38.819068 19605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:38.819658 19607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:38.820334 19599 server_base.cc:1034] running on GCE node
I20240503 08:16:38.820554 19599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:38.820842 19599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:38.822039 19599 hybrid_clock.cc:648] HybridClock initialized: now 1714724198822013 us; error 45 us; skew 500 ppm
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724198, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:38.825500 19599 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:16:38.826033 19599 webserver.cc:460] Webserver started at http://127.6.68.195:33149/ using document root <none> and password file <none>
I20240503 08:16:38.826289 19599 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:38.826370 19599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724198, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:16:38.860989 19599 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.001s sys 0.000s
I20240503 08:16:38.863797 19615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.864063 19599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20240503 08:16:38.864145 19599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "224927359bb74d81822f433620121a04"
format_stamp: "Formatted at 2024-05-03 08:16:37 on dist-test-slave-3g67"
server_key: "3ecf6513037ad77c52a4eed83e1c22da"
server_key_iv: "8637f71f5b4c2e8cee8b69a1d4457a49"
server_key_version: "kuduclusterkey@0"
I20240503 08:16:38.864553 19599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:38.884727 19599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:38.886996 19599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:38.887200 19599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:38.887419 19599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:38.887848 19599 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:38.887941 19599 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.888012 19599 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:38.888068 19599 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:38.895584 19599 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:43161
I20240503 08:16:38.896162 19599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:16:38.896561 19728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:43161 every 8 connection(s)
I20240503 08:16:38.900894 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 19599
May 03 08:16:38 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724198, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:38.911103 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:38.896994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:36605 (local address 127.6.68.254:34417)
0503 08:16:38.897049 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:38.897051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:38.901294 (+ 4243us) server_negotiation.cc:407] Connection header received
0503 08:16:38.902406 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:38.902407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:38.902420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:38.902491 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:38.903523 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.904329 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:38.905272 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.905457 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:38.907884 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:38.907888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:38.907888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:38.907895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:38.909536 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:38.910028 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:38.910029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:38.910030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:38.910054 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:38.910306 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:38.910307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:38.910307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:38.910431 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:38.910476 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:38.910897 (+ 421us) server_negotiation.cc:299] Negotiation successful
0503 08:16:38.910997 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240503 08:16:38.911581 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:38.896971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34417 (local address 127.6.68.195:36605)
0503 08:16:38.901161 (+ 4190us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:38.901207 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:38.902286 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:38.902624 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:38.902629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:38.902938 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:38.903398 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:38.903404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.904436 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:38.904436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:38.905147 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:38.905151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.905602 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:38.906045 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:38.906068 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:38.907779 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:38.909633 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:38.909641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:38.909645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:38.909939 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:38.910116 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:38.910117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:38.910117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:38.910208 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:38.910525 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:38.910530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:38.910697 (+ 167us) client_negotiation.cc:769] Sending connection context
0503 08:16:38.911227 (+ 530us) client_negotiation.cc:240] Negotiation successful
0503 08:16:38.911375 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4136,"thread_start_us":85,"threads_started":1}
I20240503 08:16:38.912205 19729 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34417
I20240503 08:16:38.912309 19729 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:38.912405 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:38.901395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60146 (local address 127.6.68.254:34417)
0503 08:16:38.901538 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:38.901540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:38.902662 (+ 1122us) server_negotiation.cc:407] Connection header received
0503 08:16:38.902683 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:38.902683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:38.902694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:38.902746 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:38.903845 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.904506 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:38.906018 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:38.906202 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:38.907229 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:38.907235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:38.907235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:38.907242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:38.909305 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:38.910620 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:38.910622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:38.910622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:38.910648 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:38.911479 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:38.911481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:38.911481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:38.911594 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:38.911933 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:38.912196 (+ 263us) server_negotiation.cc:299] Negotiation successful
0503 08:16:38.912299 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20240503 08:16:38.912544 19729 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
W20240503 08:16:38.913347 19729 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:34417 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240503 08:16:38.913417 19729 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:38.913504 19729 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
I20240503 08:16:38.913904 19729 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:38.914000 19729 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
W20240503 08:16:38.914309 19729 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
May 03 08:16:39 dist-test-slave-3g67 krb5kdc[18268](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714724199, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:39.530123 19462 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:39.530344 19462 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
I20240503 08:16:39.530732 19263 ts_manager.cc:194] Registered new tserver with Master: 4987d2a6c60e42e48c48b947361b8e17 (127.6.68.193:36615)
I20240503 08:16:39.531834 19263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:32931
I20240503 08:16:39.692241 19596 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:39.692483 19596 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
I20240503 08:16:39.692956 19263 ts_manager.cc:194] Registered new tserver with Master: 8cf6aeccea5b453ab3e97b15fbdfee9e (127.6.68.194:41835)
I20240503 08:16:39.693416 19263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:35211
May 03 08:16:39 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724199, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
I20240503 08:16:39.914500 19729 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:39.914726 19729 heartbeater.cc:511] Master 127.6.68.254:34417 requested a full tablet report, sending...
I20240503 08:16:39.915120 19263 ts_manager.cc:194] Registered new tserver with Master: 224927359bb74d81822f433620121a04 (127.6.68.195:43161)
I20240503 08:16:39.915814 19263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:36605
I20240503 08:16:39.917448 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:16:39.926195 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:39.918497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.6.68.254:34417)
0503 08:16:39.918734 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:39.918736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:39.918902 (+ 166us) server_negotiation.cc:407] Connection header received
0503 08:16:39.918926 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:39.918926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:39.918939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:39.918988 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:39.919916 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.920722 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:39.921724 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.921912 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:39.922937 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:39.922942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:39.922942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:39.922949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:39.924774 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:39.925149 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:39.925151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:39.925151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:39.925175 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:39.925336 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:39.925337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:39.925337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:39.925444 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:39.925529 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:39.925964 (+ 435us) server_negotiation.cc:299] Negotiation successful
0503 08:16:39.926064 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":62,"threads_started":1}
I20240503 08:16:39.928650 19263 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:16:39.930716 19263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:16:39.944527 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:39.938126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:43161 (local address 127.0.0.1:35800)
0503 08:16:39.940930 (+ 2804us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:39.940941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:39.940995 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:39.941306 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:39.941307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:39.941310 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:39.941514 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:39.941517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.942787 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:39.942788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:39.944192 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:39.944196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.944271 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:39.944325 (+ 54us) client_negotiation.cc:769] Sending connection context
0503 08:16:39.944385 (+ 60us) client_negotiation.cc:240] Negotiation successful
0503 08:16:39.944431 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2784,"thread_start_us":85,"threads_started":1}
I20240503 08:16:39.947875 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:39.938447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.6.68.195:43161)
0503 08:16:39.939036 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:39.939041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:39.941053 (+ 2012us) server_negotiation.cc:407] Connection header received
0503 08:16:39.941111 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:39.941113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:39.941129 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:39.941219 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:39.941603 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.942684 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:39.947051 (+ 4367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.947597 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:39.947692 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:39.947731 (+ 39us) server_negotiation.cc:299] Negotiation successful
0503 08:16:39.947775 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":51,"threads_started":1}
I20240503 08:16:39.946954 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:39.937901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:41835 (local address 127.0.0.1:51130)
0503 08:16:39.939171 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:39.939213 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:39.939288 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:39.939586 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:39.939589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:39.939590 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:39.939869 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:39.939871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.945486 (+ 5615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:39.945487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:39.946716 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:39.946720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.946793 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:39.946798 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:16:39.946832 (+ 34us) client_negotiation.cc:240] Negotiation successful
0503 08:16:39.946873 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"thread_start_us":66,"threads_started":1}
I20240503 08:16:39.950033 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:39.938333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51130 (local address 127.6.68.194:41835)
0503 08:16:39.939108 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:39.939111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:39.939368 (+ 257us) server_negotiation.cc:407] Connection header received
0503 08:16:39.939409 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:39.939411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:39.939429 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:39.939491 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:39.939954 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.945375 (+ 5421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:39.948159 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.948689 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:39.948738 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:39.948773 (+ 35us) server_negotiation.cc:299] Negotiation successful
0503 08:16:39.948814 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":54,"threads_started":1}
I20240503 08:16:39.950201 19654 tablet_service.cc:1467] Processing CreateTablet for tablet 71e0b3b70586497d8f42be675203d0c6 (DEFAULT_TABLE table=test-table [id=cf31503927e445c39d74840d9efed5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:16:39.950546 19654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e0b3b70586497d8f42be675203d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:39.950758 19530 tablet_service.cc:1467] Processing CreateTablet for tablet 71e0b3b70586497d8f42be675203d0c6 (DEFAULT_TABLE table=test-table [id=cf31503927e445c39d74840d9efed5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:16:39.951033 19530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e0b3b70586497d8f42be675203d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:39.953773 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:39.938579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:36615 (local address 127.0.0.1:45298)
0503 08:16:39.949028 (+ 10449us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:39.949042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:39.949103 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:39.950633 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:39.950634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:39.950636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:39.950836 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:39.950838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.952155 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:39.952156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:39.953119 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:39.953122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.953210 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:39.953214 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:16:39.953247 (+ 33us) client_negotiation.cc:240] Negotiation successful
0503 08:16:39.953676 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10426,"thread_start_us":62,"threads_started":1}
I20240503 08:16:39.954197 19765 tablet_bootstrap.cc:492] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e: Bootstrap starting.
I20240503 08:16:39.954679 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:39.939424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.6.68.193:36615)
0503 08:16:39.950389 (+ 10965us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:39.950394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:39.950405 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:16:39.950447 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:39.950448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:39.950466 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:39.950548 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:39.950933 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.952054 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:39.953958 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:39.954464 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:39.954518 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:39.954552 (+ 34us) server_negotiation.cc:299] Negotiation successful
0503 08:16:39.954584 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10927,"thread_start_us":56,"threads_started":1}
I20240503 08:16:39.955173 19765 tablet_bootstrap.cc:654] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:39.955359 19396 tablet_service.cc:1467] Processing CreateTablet for tablet 71e0b3b70586497d8f42be675203d0c6 (DEFAULT_TABLE table=test-table [id=cf31503927e445c39d74840d9efed5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:16:39.955593 19765 log.cc:824] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:39.955682 19396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e0b3b70586497d8f42be675203d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:39.956043 19764 tablet_bootstrap.cc:492] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: Bootstrap starting.
I20240503 08:16:39.956665 19764 tablet_bootstrap.cc:654] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:39.957005 19764 log.cc:824] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:39.957800 19765 tablet_bootstrap.cc:492] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e: No bootstrap required, opened a new log
I20240503 08:16:39.957907 19765 ts_tablet_manager.cc:1367] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:16:39.960137 19765 raft_consensus.cc:348] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4987d2a6c60e42e48c48b947361b8e17" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 36615 } } peers { permanent_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41835 } } peers { permanent_uuid: "224927359bb74d81822f433620121a04" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43161 } }
I20240503 08:16:39.960623 19765 raft_consensus.cc:374] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:39.960696 19765 raft_consensus.cc:729] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf6aeccea5b453ab3e97b15fbdfee9e, State: Initialized, Role: FOLLOWER
I20240503 08:16:39.960898 19765 consensus_queue.cc:260] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4987d2a6c60e42e48c48b947361b8e17" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 36615 } } peers { permanent_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41835 } } peers { permanent_uuid: "224927359bb74d81822f433620121a04" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43161 } }
I20240503 08:16:39.961256 19765 ts_tablet_manager.cc:1398] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:16:39.961412 19596 heartbeater.cc:503] Master 127.6.68.254:34417 was elected leader, sending a full tablet report...
I20240503 08:16:39.962327 19764 tablet_bootstrap.cc:492] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: No bootstrap required, opened a new log
I20240503 08:16:39.962448 19764 ts_tablet_manager.cc:1367] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:16:39.964638 19764 raft_consensus.cc:348] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4987d2a6c60e42e48c48b947361b8e17" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 36615 } } peers { permanent_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41835 } } peers { permanent_uuid: "224927359bb74d81822f433620121a04" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43161 } }
I20240503 08:16:39.964879 19764 raft_consensus.cc:374] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:39.964943 19764 raft_consensus.cc:729] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224927359bb74d81822f433620121a04, State: Initialized, Role: FOLLOWER
I20240503 08:16:39.965534 19764 consensus_queue.cc:260] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4987d2a6c60e42e48c48b947361b8e17" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 36615 } } peers { permanent_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41835 } } peers { permanent_uuid: "224927359bb74d81822f433620121a04" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43161 } }
I20240503 08:16:39.965934 19764 ts_tablet_manager.cc:1398] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:16:39.966316 19729 heartbeater.cc:503] Master 127.6.68.254:34417 was elected leader, sending a full tablet report...
I20240503 08:16:39.966972 19768 tablet_bootstrap.cc:492] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17: Bootstrap starting.
I20240503 08:16:39.967729 19768 tablet_bootstrap.cc:654] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:39.968087 19768 log.cc:824] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:39.969031 19768 tablet_bootstrap.cc:492] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17: No bootstrap required, opened a new log
I20240503 08:16:39.969115 19768 ts_tablet_manager.cc:1367] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:16:39.971163 19768 raft_consensus.cc:348] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4987d2a6c60e42e48c48b947361b8e17" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 36615 } } peers { permanent_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41835 } } peers { permanent_uuid: "224927359bb74d81822f433620121a04" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43161 } }
I20240503 08:16:39.971474 19768 raft_consensus.cc:374] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:39.971518 19768 raft_consensus.cc:729] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4987d2a6c60e42e48c48b947361b8e17, State: Initialized, Role: FOLLOWER
I20240503 08:16:39.971616 19768 consensus_queue.cc:260] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4987d2a6c60e42e48c48b947361b8e17" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 36615 } } peers { permanent_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41835 } } peers { permanent_uuid: "224927359bb74d81822f433620121a04" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43161 } }
I20240503 08:16:39.972173 19768 ts_tablet_manager.cc:1398] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:16:39.972359 19462 heartbeater.cc:503] Master 127.6.68.254:34417 was elected leader, sending a full tablet report...
W20240503 08:16:39.998015 19463 tablet.cc:2369] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:16:40.147606 19730 tablet.cc:2369] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: Can't schedule compaction. Clean time has not been advanced past its initial value.
May 03 08:16:40 dist-test-slave-3g67 krb5kdc[18268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724199, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for HTTP/127.6.68.212@KRBTEST.COM
W20240503 08:16:40.171950 19597 tablet.cc:2369] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:16:40.204744 19772 raft_consensus.cc:482] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:16:40.204886 19772 raft_consensus.cc:504] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4987d2a6c60e42e48c48b947361b8e17" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 36615 } } peers { permanent_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41835 } } peers { permanent_uuid: "224927359bb74d81822f433620121a04" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43161 } }
I20240503 08:16:40.205250 19772 leader_election.cc:288] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cf6aeccea5b453ab3e97b15fbdfee9e (127.6.68.194:41835), 224927359bb74d81822f433620121a04 (127.6.68.195:43161)
I20240503 08:16:40.209759 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:40.205635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:37171 (local address 127.6.68.194:41835)
0503 08:16:40.205681 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:40.205684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:40.206147 (+ 463us) server_negotiation.cc:407] Connection header received
0503 08:16:40.206255 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:40.206256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:40.206265 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:40.206293 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:40.206824 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:40.207602 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:40.209011 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:40.209485 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:40.209497 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:40.209626 (+ 129us) server_negotiation.cc:299] Negotiation successful
0503 08:16:40.209670 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:16:40.210220 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:40.205427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:41835 (local address 127.6.68.193:37171)
0503 08:16:40.206076 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:40.206087 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:40.206177 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:40.206368 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:40.206369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:40.206371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:40.206710 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:40.206724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:40.207751 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:40.207752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:40.208907 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:40.208910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:40.209531 (+ 621us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:40.209535 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:16:40.210088 (+ 553us) client_negotiation.cc:240] Negotiation successful
0503 08:16:40.210135 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":97,"threads_started":1}
I20240503 08:16:40.210721 19550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e0b3b70586497d8f42be675203d0c6" candidate_uuid: "4987d2a6c60e42e48c48b947361b8e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" is_pre_election: true
I20240503 08:16:40.210927 19550 raft_consensus.cc:2454] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4987d2a6c60e42e48c48b947361b8e17 in term 0.
I20240503 08:16:40.211186 19351 leader_election.cc:302] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4987d2a6c60e42e48c48b947361b8e17, 8cf6aeccea5b453ab3e97b15fbdfee9e; no voters:
I20240503 08:16:40.211321 19772 raft_consensus.cc:2781] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:16:40.211373 19772 raft_consensus.cc:482] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:16:40.211396 19772 raft_consensus.cc:3037] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:40.212303 19772 raft_consensus.cc:504] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4987d2a6c60e42e48c48b947361b8e17" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 36615 } } peers { permanent_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41835 } } peers { permanent_uuid: "224927359bb74d81822f433620121a04" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43161 } }
I20240503 08:16:40.212442 19772 leader_election.cc:288] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [CANDIDATE]: Term 1 election: Requested vote from peers 8cf6aeccea5b453ab3e97b15fbdfee9e (127.6.68.194:41835), 224927359bb74d81822f433620121a04 (127.6.68.195:43161)
I20240503 08:16:40.212777 19550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e0b3b70586497d8f42be675203d0c6" candidate_uuid: "4987d2a6c60e42e48c48b947361b8e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e"
I20240503 08:16:40.212858 19550 raft_consensus.cc:3037] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:40.213726 19550 raft_consensus.cc:2454] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4987d2a6c60e42e48c48b947361b8e17 in term 1.
I20240503 08:16:40.213959 19351 leader_election.cc:302] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4987d2a6c60e42e48c48b947361b8e17, 8cf6aeccea5b453ab3e97b15fbdfee9e; no voters:
I20240503 08:16:40.214089 19772 raft_consensus.cc:2781] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:16:40.214259 19772 raft_consensus.cc:686] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [term 1 LEADER]: Becoming Leader. State: Replica: 4987d2a6c60e42e48c48b947361b8e17, State: Running, Role: LEADER
I20240503 08:16:40.214354 19772 consensus_queue.cc:237] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4987d2a6c60e42e48c48b947361b8e17" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 36615 } } peers { permanent_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41835 } } peers { permanent_uuid: "224927359bb74d81822f433620121a04" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43161 } }
I20240503 08:16:40.215119 19263 catalog_manager.cc:5484] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4987d2a6c60e42e48c48b947361b8e17 (127.6.68.193). New cstate: current_term: 1 leader_uuid: "4987d2a6c60e42e48c48b947361b8e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4987d2a6c60e42e48c48b947361b8e17" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 36615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "224927359bb74d81822f433620121a04" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43161 } health_report { overall_health: UNKNOWN } } }
I20240503 08:16:40.218801 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:40.205832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:54509 (local address 127.6.68.195:43161)
0503 08:16:40.206010 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:40.206012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:40.210673 (+ 4661us) server_negotiation.cc:407] Connection header received
0503 08:16:40.210782 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:40.210783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:40.210793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:40.210829 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:40.215184 (+ 4355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:40.215963 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:40.218037 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:40.218537 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:40.218548 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:40.218673 (+ 125us) server_negotiation.cc:299] Negotiation successful
0503 08:16:40.218726 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20240503 08:16:40.219313 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:40.205486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:43161 (local address 127.6.68.193:54509)
0503 08:16:40.210622 (+ 5136us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:40.210632 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:40.210700 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:40.214855 (+ 4155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:40.214857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:40.214859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:40.215068 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:40.215069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:40.216959 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:40.216960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:40.217884 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:40.217887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:40.218580 (+ 693us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:40.218585 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:16:40.219190 (+ 605us) client_negotiation.cc:240] Negotiation successful
0503 08:16:40.219241 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5122,"thread_start_us":104,"threads_started":1}
I20240503 08:16:40.219800 19672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e0b3b70586497d8f42be675203d0c6" candidate_uuid: "4987d2a6c60e42e48c48b947361b8e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "224927359bb74d81822f433620121a04" is_pre_election: true
I20240503 08:16:40.219961 19672 raft_consensus.cc:2454] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4987d2a6c60e42e48c48b947361b8e17 in term 0.
I20240503 08:16:40.220106 19671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e0b3b70586497d8f42be675203d0c6" candidate_uuid: "4987d2a6c60e42e48c48b947361b8e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "224927359bb74d81822f433620121a04"
I20240503 08:16:40.220162 19671 raft_consensus.cc:3037] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:40.221140 19671 raft_consensus.cc:2454] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4987d2a6c60e42e48c48b947361b8e17 in term 1.
I20240503 08:16:40.224843 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:40.221970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45300 (local address 127.6.68.193:36615)
0503 08:16:40.222009 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:40.222010 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:40.222020 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:16:40.222084 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:40.222085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:40.222104 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:40.222182 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:16:40.222542 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:40.223248 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:40.224154 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:40.224335 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:40.224380 (+ 45us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:16:40.224491 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:16:40.224563 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:40.224746 (+ 183us) server_negotiation.cc:299] Negotiation successful
0503 08:16:40.224768 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:16:40.226810 19671 raft_consensus.cc:1261] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04 [term 1 FOLLOWER]: Refusing update from remote peer 4987d2a6c60e42e48c48b947361b8e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:16:40.226905 19550 raft_consensus.cc:1261] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e [term 1 FOLLOWER]: Refusing update from remote peer 4987d2a6c60e42e48c48b947361b8e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:16:40.227169 19777 consensus_queue.cc:1035] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cf6aeccea5b453ab3e97b15fbdfee9e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:16:40.227346 19772 consensus_queue.cc:1035] T 71e0b3b70586497d8f42be675203d0c6 P 4987d2a6c60e42e48c48b947361b8e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "224927359bb74d81822f433620121a04" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:16:40.245290 19780 mvcc.cc:205] Tried to move back new op lower bound from 7023510324126683136 to 7023510324078534656. Current Snapshot: MvccSnapshot[applied={T|T < 7023510324126683136}]
I20240503 08:16:40.249966 19263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44090:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:16:40.250082 19263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44090:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:16:40.251053 19263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b7528708cd11492aa571a3057ede9400: Sending DeleteTablet for 3 replicas of tablet 71e0b3b70586497d8f42be675203d0c6
I20240503 08:16:40.251536 19530 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e0b3b70586497d8f42be675203d0c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:16:40 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:51130
I20240503 08:16:40.252044 19396 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e0b3b70586497d8f42be675203d0c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:16:40 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:45298
I20240503 08:16:40.252624 19654 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e0b3b70586497d8f42be675203d0c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:16:40 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:35800
I20240503 08:16:40.253109 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 19302
I20240503 08:16:40.257659 19781 mvcc.cc:205] Tried to move back new op lower bound from 7023510324126683136 to 7023510324078534656. Current Snapshot: MvccSnapshot[applied={T|T < 7023510324126683136 or (T in {7023510324126683136})}]
I20240503 08:16:40.263767 19790 tablet_replica.cc:330] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e: stopping tablet replica
I20240503 08:16:40.263803 19792 tablet_replica.cc:330] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: stopping tablet replica
I20240503 08:16:40.263932 19792 raft_consensus.cc:2229] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:16:40.264075 19792 raft_consensus.cc:2258] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:16:40.264662 19792 ts_tablet_manager.cc:1875] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:16:40.270267 19792 ts_tablet_manager.cc:1888] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:16:40.270360 19792 log.cc:1196] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/71e0b3b70586497d8f42be675203d0c6
I20240503 08:16:40.270632 19792 ts_tablet_manager.cc:1909] T 71e0b3b70586497d8f42be675203d0c6 P 224927359bb74d81822f433620121a04: Deleting consensus metadata
I20240503 08:16:40.271018 19790 raft_consensus.cc:2229] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:16:40.271066 19248 catalog_manager.cc:4830] TS 224927359bb74d81822f433620121a04 (127.6.68.195:43161): tablet 71e0b3b70586497d8f42be675203d0c6 (table test-table [id=cf31503927e445c39d74840d9efed5da]) successfully deleted
I20240503 08:16:40.271236 19790 raft_consensus.cc:2258] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:16:40.271479 19251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:36615 (error 108)
I20240503 08:16:40.272230 19790 ts_tablet_manager.cc:1875] T 71e0b3b70586497d8f42be675203d0c6 P 8cf6aeccea5b453ab3e97b15fbdfee9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:16:40.272600 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 19466
I20240503 08:16:40.284651 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:40.284477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:36615 (local address 127.0.0.1:45308)
0503 08:16:40.284540 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:40.284596 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:36615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20240503 08:16:40.284638 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 19599
I20240503 08:16:40.284790 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:40.284682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:41835 (local address 127.0.0.1:51144)
0503 08:16:40.284753 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:40.284775 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.194:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20240503 08:16:40.285341 19251 catalog_manager.cc:4573] TS 4987d2a6c60e42e48c48b947361b8e17 (127.6.68.193:36615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71e0b3b70586497d8f42be675203d0c6: Network error: Client connection negotiation failed: client connection to 127.6.68.193:36615: connect: Connection refused (error 111)
I20240503 08:16:40.291957 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 19231
I20240503 08:16:40.298226 6419 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20240503 08:16:40.643023 6419 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20240503 08:16:40.643165 6419 mini_ranger.cc:66] Stopping Ranger...
I20240503 08:16:40.988765 6419 mini_ranger.cc:68] Stopped Ranger
2024-05-03 08:16:40.988 UTC [18298] LOG: received smart shutdown request
2024-05-03 08:16:40.993 UTC [18298] LOG: background worker "logical replication launcher" (PID 18307) exited with exit code 1
2024-05-03 08:16:40.993 UTC [18302] LOG: shutting down
2024-05-03 08:16:41.197 UTC [18298] LOG: database system is shut down
2024-05-03T08:16:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45857 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:16:41 dist-test-slave-3g67 krb5kdc[19802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:16:41 dist-test-slave-3g67 krb5kdc[19802](info): set up 2 sockets
May 03 08:16:41 dist-test-slave-3g67 krb5kdc[19802](info): commencing operation
krb5kdc: starting...
W20240503 08:16:42.271907 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:16:42.395447 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40729
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40729,127.6.68.253:38441,127.6.68.252:43905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:42.476361 19815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:42.476565 19815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:42.476588 19815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:42.477901 19815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:16:42.477947 19815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:42.477962 19815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:42.477973 19815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:16:42.477985 19815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:16:42.479667 19815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40729,127.6.68.253:38441,127.6.68.252:43905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:42.479902 19815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:42.480089 19815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:42.484752 19820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:42.485134 19821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:42.485795 19823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:42.486323 19815 server_base.cc:1034] running on GCE node
I20240503 08:16:42.486560 19815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:42.486811 19815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:42.487968 19815 hybrid_clock.cc:648] HybridClock initialized: now 1714724202487953 us; error 29 us; skew 500 ppm
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:42.496351 19815 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:16:42.496785 19815 webserver.cc:460] Webserver started at http://127.6.68.254:36003/ using document root <none> and password file <none>
I20240503 08:16:42.496968 19815 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:42.497009 19815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:42.497130 19815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:16:42.498224 19815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "32daed9e1bb248888a7d16e41e963472"
format_stamp: "Formatted at 2024-05-03 08:16:42 on dist-test-slave-3g67"
server_key: "6a0998064dad18f16dbba36c0d87771d"
server_key_iv: "7f792ede6eb5b3168f8d225c73c2a73d"
server_key_version: "encryptionkey@0"
I20240503 08:16:42.498548 19815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "32daed9e1bb248888a7d16e41e963472"
format_stamp: "Formatted at 2024-05-03 08:16:42 on dist-test-slave-3g67"
server_key: "6a0998064dad18f16dbba36c0d87771d"
server_key_iv: "7f792ede6eb5b3168f8d225c73c2a73d"
server_key_version: "encryptionkey@0"
I20240503 08:16:42.499876 19815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240503 08:16:42.500789 19830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:42.501122 19815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:42.501250 19815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "32daed9e1bb248888a7d16e41e963472"
format_stamp: "Formatted at 2024-05-03 08:16:42 on dist-test-slave-3g67"
server_key: "6a0998064dad18f16dbba36c0d87771d"
server_key_iv: "7f792ede6eb5b3168f8d225c73c2a73d"
server_key_version: "encryptionkey@0"
I20240503 08:16:42.501328 19815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:42.517204 19815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:42.519428 19815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:42.519572 19815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:42.523599 19815 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:40729
I20240503 08:16:42.523672 19882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:40729 every 8 connection(s)
I20240503 08:16:42.524011 19815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:16:42.524621 19883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:42.526490 19883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:16:42.530745 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 19815
I20240503 08:16:42.530872 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:16:42.531076 6419 external_mini_cluster.cc:1428] Setting key 4023b22c678732db4791894627ad5d37
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:42.536548 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.527190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40729 (local address 127.0.0.1:59246)
0503 08:16:42.527415 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.527440 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:42.527962 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:42.528234 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:42.528237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:42.528550 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:42.528797 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.528800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.529658 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.529660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:42.530042 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.530044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.530168 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.530840 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:42.530857 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:42.532790 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:42.535082 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.535088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.535091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.535272 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.535505 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.535506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.535506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.535552 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.536038 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:42.536044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:42.536116 (+ 72us) client_negotiation.cc:769] Sending connection context
0503 08:16:42.536258 (+ 142us) client_negotiation.cc:240] Negotiation successful
0503 08:16:42.536391 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":124,"threads_started":1}
I20240503 08:16:42.536636 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.527242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.6.68.254:40729)
0503 08:16:42.527433 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.527445 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.527543 (+ 98us) server_negotiation.cc:407] Connection header received
0503 08:16:42.528095 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.528097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.528114 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.528192 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.528883 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.529562 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.530122 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.530273 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.532894 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.532905 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.532905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.532912 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.534978 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.535369 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.535372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.535373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.535434 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.535611 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.535612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.535613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.535931 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.536039 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.536297 (+ 258us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.536492 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20240503 08:16:42.537425 19883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:16:42.537835 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.537690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:38441 (local address 127.0.0.1:44270)
0503 08:16:42.537760 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.537809 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20240503 08:16:42.537971 19832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.68.253:38441: connect: Connection refused (error 111)
I20240503 08:16:42.538429 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.538338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:38441 (local address 127.0.0.1:44272)
0503 08:16:42.538384 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.538409 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20240503 08:16:42.538535 19883 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.253:38441: Network error: Client connection negotiation failed: client connection to 127.6.68.253:38441: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.6.68.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab.
Entry for principal kudu/127.6.68.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab.
I20240503 08:16:42.559885 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.253:38441
--webserver_interface=127.6.68.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab
--principal=kudu/127.6.68.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40729,127.6.68.253:38441,127.6.68.252:43905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:16:42.587743 19883 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } attempt: 1
I20240503 08:16:42.588354 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.588201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:38441 (local address 127.0.0.1:44280)
0503 08:16:42.588271 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.588318 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20240503 08:16:42.588701 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.588609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:38441 (local address 127.0.0.1:44290)
0503 08:16:42.588655 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.588681 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240503 08:16:42.588821 19883 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.253:38441: Network error: Client connection negotiation failed: client connection to 127.6.68.253:38441: connect: Connection refused (error 111)
W20240503 08:16:42.639928 19890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:42.640126 19890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:42.640149 19890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:42.641552 19890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:16:42.641605 19890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:42.641619 19890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:42.641631 19890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:16:42.641645 19890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:16:42.643322 19890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40729,127.6.68.253:38441,127.6.68.252:43905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab
--principal=kudu/127.6.68.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.253:38441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.68.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:42.643527 19890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:42.643749 19890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:42.648344 19896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:42.648731 19895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:42.649438 19898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:42.649994 19890 server_base.cc:1034] running on GCE node
I20240503 08:16:42.650225 19890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:42.650492 19890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:42.651665 19890 hybrid_clock.cc:648] HybridClock initialized: now 1714724202651648 us; error 45 us; skew 500 ppm
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:42.654067 19890 init.cc:377] Logged in from keytab as kudu/127.6.68.253@KRBTEST.COM (short username kudu)
I20240503 08:16:42.654507 19890 webserver.cc:460] Webserver started at http://127.6.68.253:46737/ using document root <none> and password file <none>
I20240503 08:16:42.654685 19890 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:42.654729 19890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:42.654932 19890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:16:42.656103 19890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/data/instance:
uuid: "83a043ef94d94534aa0cdd0ba44b2227"
format_stamp: "Formatted at 2024-05-03 08:16:42 on dist-test-slave-3g67"
server_key: "ae757b9e03952f4aa5800178376a728d"
server_key_iv: "4af784fa912f1e456e5258c7df5c8187"
server_key_version: "encryptionkey@0"
I20240503 08:16:42.656446 19890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/wal/instance:
uuid: "83a043ef94d94534aa0cdd0ba44b2227"
format_stamp: "Formatted at 2024-05-03 08:16:42 on dist-test-slave-3g67"
server_key: "ae757b9e03952f4aa5800178376a728d"
server_key_iv: "4af784fa912f1e456e5258c7df5c8187"
server_key_version: "encryptionkey@0"
I20240503 08:16:42.657004 19883 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } attempt: 2
I20240503 08:16:42.657450 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.657282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:38441 (local address 127.0.0.1:44296)
0503 08:16:42.657354 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.657395 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20240503 08:16:42.657966 19890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:42.658087 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.657908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:38441 (local address 127.0.0.1:44306)
0503 08:16:42.657979 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.658038 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20240503 08:16:42.658200 19883 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.253:38441: Network error: Client connection negotiation failed: client connection to 127.6.68.253:38441: connect: Connection refused (error 111)
I20240503 08:16:42.659061 19905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:42.659300 19890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:42.659410 19890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/wal
uuid: "83a043ef94d94534aa0cdd0ba44b2227"
format_stamp: "Formatted at 2024-05-03 08:16:42 on dist-test-slave-3g67"
server_key: "ae757b9e03952f4aa5800178376a728d"
server_key_iv: "4af784fa912f1e456e5258c7df5c8187"
server_key_version: "encryptionkey@0"
I20240503 08:16:42.659485 19890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:42.678894 19890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:42.681658 19890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:42.681797 19890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:42.685910 19890 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.253:38441
I20240503 08:16:42.686040 19957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.253:38441 every 8 connection(s)
I20240503 08:16:42.686614 19890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
I20240503 08:16:42.687357 19958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:42.689209 19958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:42.694780 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 19890
I20240503 08:16:42.694880 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-1/wal/instance
I20240503 08:16:42.695092 6419 external_mini_cluster.cc:1428] Setting key 845f51b429bf05608faa2b521d4058a7
I20240503 08:16:42.697994 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.689916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.6.68.254:40729)
0503 08:16:42.690010 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.690012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.690187 (+ 175us) server_negotiation.cc:407] Connection header received
0503 08:16:42.690676 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.690677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.690690 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.690752 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.691501 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.692013 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.692564 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.692764 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.695035 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.695039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.695039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.695046 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.696734 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.697140 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.697141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.697141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.697164 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.697370 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.697371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.697372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.697479 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.697543 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.697801 (+ 258us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.697898 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20240503 08:16:42.698182 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.689853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40729 (local address 127.0.0.1:59254)
0503 08:16:42.690072 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.690094 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:42.690564 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:42.690810 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:42.690813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:42.691117 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:42.691406 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.691410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.692110 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.692112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:42.692489 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.692491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.692624 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.693265 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:42.693281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:42.694935 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:42.696828 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.696835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.696837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.697063 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.697219 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.697220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.697220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.697276 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.697541 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:42.697547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:42.697671 (+ 124us) client_negotiation.cc:769] Sending connection context
0503 08:16:42.697772 (+ 101us) client_negotiation.cc:240] Negotiation successful
0503 08:16:42.697927 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20240503 08:16:42.698801 19958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.68.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.252@KRBTEST.COM" created.
I20240503 08:16:42.706286 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.699138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.6.68.253:38441)
0503 08:16:42.699287 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.699293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.699308 (+ 15us) server_negotiation.cc:407] Connection header received
0503 08:16:42.699330 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.699332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.699343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.699402 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.699921 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.700574 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.701323 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.701464 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.703210 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.703219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.703220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.703225 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.704876 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.705222 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.705225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.705225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.705258 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.705410 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.705410 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.705411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.705520 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.705604 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.705962 (+ 358us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.706062 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20240503 08:16:42.706355 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.699076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:38441 (local address 127.0.0.1:44316)
0503 08:16:42.699130 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.699142 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:42.699189 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:42.699420 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:42.699420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:42.699596 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:42.699843 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.699845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.700660 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.700661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:42.701227 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.701229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.701321 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.701804 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:42.701808 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:42.703126 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:42.704971 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.704972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.704972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.705110 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.705326 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.705327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.705327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.705368 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.705663 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:42.705665 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:42.705756 (+ 91us) client_negotiation.cc:769] Sending connection context
0503 08:16:42.705904 (+ 148us) client_negotiation.cc:240] Negotiation successful
0503 08:16:42.706121 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":10}
I20240503 08:16:42.708776 19958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:16:42.709175 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.709016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41020)
0503 08:16:42.709068 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.709124 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240503 08:16:42.709321 19907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
I20240503 08:16:42.709895 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.709780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41024)
0503 08:16:42.709847 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.709874 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20240503 08:16:42.710013 19958 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:43905: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab.
Entry for principal kudu/127.6.68.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab.
I20240503 08:16:42.723358 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.252:43905
--webserver_interface=127.6.68.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab
--principal=kudu/127.6.68.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40729,127.6.68.253:38441,127.6.68.252:43905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:16:42.749383 19883 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } attempt: 3
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
I20240503 08:16:42.756932 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.749904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.6.68.253:38441)
0503 08:16:42.749978 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.749980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.749992 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:16:42.750054 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.750055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.750071 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.750135 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.750861 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.751590 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.752213 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.752394 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.754213 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.754235 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.754235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.754243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.755714 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.756206 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.756207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.756208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.756237 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.756432 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.756433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.756433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.756539 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.756605 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.756773 (+ 168us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.756855 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240503 08:16:42.756958 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.749840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:38441 (local address 127.0.0.1:44332)
0503 08:16:42.749921 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.749935 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:42.749997 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:42.750245 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:42.750247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:42.750552 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:42.750746 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.750754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.751731 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.751733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:42.752137 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.752139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.752203 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.752756 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:42.752761 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:42.754108 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:42.755897 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.755899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.755900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.756097 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.756313 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.756314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.756314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.756347 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.756615 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:42.756616 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:42.756674 (+ 58us) client_negotiation.cc:769] Sending connection context
0503 08:16:42.756746 (+ 72us) client_negotiation.cc:240] Negotiation successful
0503 08:16:42.756861 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20240503 08:16:42.757558 19883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:16:42.757928 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.757819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41030)
0503 08:16:42.757870 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.757902 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20240503 08:16:42.758235 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.758156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41044)
0503 08:16:42.758194 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.758217 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240503 08:16:42.758319 19883 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:43905: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
I20240503 08:16:42.759198 19958 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } attempt: 1
I20240503 08:16:42.759544 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.759448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41050)
0503 08:16:42.759499 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.759522 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20240503 08:16:42.759881 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.759792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41062)
0503 08:16:42.759832 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.759863 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240503 08:16:42.759975 19958 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:43905: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
I20240503 08:16:42.789506 19883 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } attempt: 1
I20240503 08:16:42.790014 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.789894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41068)
0503 08:16:42.789949 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.789983 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20240503 08:16:42.790411 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.790250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41070)
0503 08:16:42.790320 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.790370 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20240503 08:16:42.790537 19883 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:43905: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
W20240503 08:16:42.808161 19965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:42.808362 19965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:42.808385 19965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:42.809741 19965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:16:42.809798 19965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:42.809811 19965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:42.809824 19965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:16:42.809835 19965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:16:42.811556 19965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40729,127.6.68.253:38441,127.6.68.252:43905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab
--principal=kudu/127.6.68.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.252:43905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.68.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:42.811817 19965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:42.811997 19965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:42.816459 19970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:42.817171 19971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:42.817693 19973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:42.818511 19965 server_base.cc:1034] running on GCE node
I20240503 08:16:42.818815 19965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:42.819087 19965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:42.820246 19965 hybrid_clock.cc:648] HybridClock initialized: now 1714724202820231 us; error 29 us; skew 500 ppm
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:42.822706 19965 init.cc:377] Logged in from keytab as kudu/127.6.68.252@KRBTEST.COM (short username kudu)
I20240503 08:16:42.823158 19965 webserver.cc:460] Webserver started at http://127.6.68.252:38253/ using document root <none> and password file <none>
I20240503 08:16:42.823357 19965 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:42.823405 19965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:42.823519 19965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:16:42.824637 19965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/data/instance:
uuid: "070893ff3ab84ece8a300f3ef66a5f97"
format_stamp: "Formatted at 2024-05-03 08:16:42 on dist-test-slave-3g67"
server_key: "e86df009cb77cf31cd8aa2ddd32022e9"
server_key_iv: "c20183ee65f339952ef06427c634a2ba"
server_key_version: "encryptionkey@0"
I20240503 08:16:42.824956 19965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/wal/instance:
uuid: "070893ff3ab84ece8a300f3ef66a5f97"
format_stamp: "Formatted at 2024-05-03 08:16:42 on dist-test-slave-3g67"
server_key: "e86df009cb77cf31cd8aa2ddd32022e9"
server_key_iv: "c20183ee65f339952ef06427c634a2ba"
server_key_version: "encryptionkey@0"
I20240503 08:16:42.826289 19965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:16:42.827096 19980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:42.827308 19965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:42.827390 19965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/wal
uuid: "070893ff3ab84ece8a300f3ef66a5f97"
format_stamp: "Formatted at 2024-05-03 08:16:42 on dist-test-slave-3g67"
server_key: "e86df009cb77cf31cd8aa2ddd32022e9"
server_key_iv: "c20183ee65f339952ef06427c634a2ba"
server_key_version: "encryptionkey@0"
I20240503 08:16:42.827464 19965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:42.828132 19958 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } attempt: 2
I20240503 08:16:42.828574 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.828442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41074)
0503 08:16:42.828509 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.828545 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20240503 08:16:42.828945 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.828862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41078)
0503 08:16:42.828905 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.828927 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240503 08:16:42.829037 19958 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:43905: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43905: connect: Connection refused (error 111)
I20240503 08:16:42.844678 19965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:42.846712 19965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:42.846822 19965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:42.850653 19965 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.252:43905
I20240503 08:16:42.850708 20032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.252:43905 every 8 connection(s)
I20240503 08:16:42.851006 19965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
I20240503 08:16:42.851716 20033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:42.853569 20033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:16:42.858336 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 19965
I20240503 08:16:42.858438 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714724013808011-6419-0/minicluster-data/master-2/wal/instance
I20240503 08:16:42.858615 6419 external_mini_cluster.cc:1428] Setting key c247da23e15de51be7a088f7f90a08c3
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:42.862639 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.854291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.6.68.254:40729)
0503 08:16:42.854392 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.854395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.854557 (+ 162us) server_negotiation.cc:407] Connection header received
0503 08:16:42.855083 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.855084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.855096 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.855146 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.855972 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.856452 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.857004 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.857200 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.859470 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.859474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.859475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.859483 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.861241 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.861707 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.861708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.861708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.861730 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.861936 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.861937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.861937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.862044 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.862106 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.862451 (+ 345us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.862557 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20240503 08:16:42.862792 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.854226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40729 (local address 127.0.0.1:59270)
0503 08:16:42.854443 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.854469 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:42.854972 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:42.855228 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:42.855234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:42.855539 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:42.855881 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.855886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.856549 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.856550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:42.856928 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.856929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.857062 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.857695 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:42.857712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:42.859368 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:42.861337 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.861343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.861346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.861624 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.861809 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.861809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.861810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.861879 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.862133 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:42.862140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:42.862310 (+ 170us) client_negotiation.cc:769] Sending connection context
0503 08:16:42.862410 (+ 100us) client_negotiation.cc:240] Negotiation successful
0503 08:16:42.862559 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
I20240503 08:16:42.863665 20033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
I20240503 08:16:42.865689 19883 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } attempt: 2
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
I20240503 08:16:42.867834 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.859029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.6.68.252:43905)
0503 08:16:42.859331 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.859338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.859350 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:16:42.859369 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.859370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.859381 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.859444 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.860366 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.861428 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.862098 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.862264 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.864767 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.864776 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.864776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.864783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.866672 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.867022 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.867025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.867025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.867052 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.867230 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.867231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.867232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.867339 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.867491 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.867565 (+ 74us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.867682 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":71,"threads_started":1}
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
I20240503 08:16:42.870519 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.863930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:38441 (local address 127.0.0.1:44340)
0503 08:16:42.863988 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.863999 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:42.864048 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:42.864173 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:42.864174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:42.864319 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:42.864486 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.864488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.865064 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.865065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:42.865422 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.865423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.865486 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.866022 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:42.866025 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:42.867359 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:42.869527 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.869528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.869528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.869739 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.869933 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.869933 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.869934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.869962 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.870197 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:42.870198 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:42.870244 (+ 46us) client_negotiation.cc:769] Sending connection context
0503 08:16:42.870318 (+ 74us) client_negotiation.cc:240] Negotiation successful
0503 08:16:42.870416 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20240503 08:16:42.870527 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.863997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.6.68.253:38441)
0503 08:16:42.864056 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.864058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.864069 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:16:42.864098 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.864099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.864107 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.864149 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.864566 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.864997 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.865492 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.865679 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.867454 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.867458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.867458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.867466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.869447 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.869834 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.869836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.869836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.869864 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.870022 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.870023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.870023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.870141 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.870207 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.870348 (+ 141us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.870436 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:16:42.871243 20033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:16:42.873332 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.865938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41100)
0503 08:16:42.866014 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.866033 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:42.866087 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:42.866512 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:42.866513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:42.866635 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:42.866781 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.866783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.867696 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.867698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:42.868287 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.868290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.868340 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.868922 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:42.868926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:42.870386 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:42.872501 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.872502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.872502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.872645 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.872809 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.872810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.872811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.872839 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.873026 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:42.873027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:42.873071 (+ 44us) client_negotiation.cc:769] Sending connection context
0503 08:16:42.873137 (+ 66us) client_negotiation.cc:240] Negotiation successful
0503 08:16:42.873242 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20240503 08:16:42.873517 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.866162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.6.68.252:43905)
0503 08:16:42.866341 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.866343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.866355 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:16:42.866376 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.866376 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.866389 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.866487 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.866872 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.867563 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.868521 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.868729 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.870467 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.870470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.870471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.870476 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.872430 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.872708 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.872709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.872709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.872752 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.872890 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.872891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.872892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.872973 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.873030 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.873160 (+ 130us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.873238 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:42.875267 19883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472: Bootstrap starting.
I20240503 08:16:42.876024 19883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472: Neither blocks nor log segments found. Creating new log.
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
I20240503 08:16:42.876345 19883 log.cc:824] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:42.877321 19883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472: No bootstrap required, opened a new log
I20240503 08:16:42.877333 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.868597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.6.68.254:40729)
0503 08:16:42.868635 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.868637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.868645 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:16:42.868665 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.868666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.868675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.868733 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.870143 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.870752 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.871479 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.871665 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.874613 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.874617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.874617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.874624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.876264 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.876621 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.876623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.876623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.876650 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.876837 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.876839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.876839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.876933 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.877059 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.877155 (+ 96us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.877251 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240503 08:16:42.879076 19883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } }
I20240503 08:16:42.879194 19883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:42.879215 19883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32daed9e1bb248888a7d16e41e963472, State: Initialized, Role: FOLLOWER
I20240503 08:16:42.879319 19883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } }
I20240503 08:16:42.879546 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.872172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41110)
0503 08:16:42.872222 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.872231 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:42.872270 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:42.872441 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:42.872441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:42.872627 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:42.872825 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.872827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.873586 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.873587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:42.874106 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.874110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.874174 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.874600 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:42.874605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:42.876395 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:42.878597 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.878598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.878598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.878782 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.878969 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.878970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.878970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.878999 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.879212 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:42.879213 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:42.879251 (+ 38us) client_negotiation.cc:769] Sending connection context
0503 08:16:42.879317 (+ 66us) client_negotiation.cc:240] Negotiation successful
0503 08:16:42.879426 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20240503 08:16:42.879714 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.872225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.6.68.252:43905)
0503 08:16:42.872260 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.872262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.872270 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:16:42.872341 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.872342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.872353 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.872422 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.872916 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.873512 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.874305 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.874486 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.876493 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.876497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.876497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.876503 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.878512 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.878862 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.878863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.878864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.878886 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.879052 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.879052 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.879052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.879149 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.879214 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.879371 (+ 157us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.879478 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":22}
I20240503 08:16:42.879607 19883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:16:42.879695 20040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } } }
I20240503 08:16:42.880421 20040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 08:16:42.882174 20051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:16:42.882290 20051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
I20240503 08:16:42.883069 20033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97: Bootstrap starting.
I20240503 08:16:42.883755 20033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:42.884050 20033 log.cc:824] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:42.884837 20033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97: No bootstrap required, opened a new log
I20240503 08:16:42.886618 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.878046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.6.68.253:38441)
0503 08:16:42.878098 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.878100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.878109 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:16:42.878194 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.878195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.878206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.878279 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.878963 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.879592 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.880406 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.880586 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.883342 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.883346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.883346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.883354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.885232 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.885598 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.885599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.885599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.885622 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.885772 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.885773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.885773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.885886 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.886028 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.886435 (+ 407us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.886525 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240503 08:16:42.883282 19815 master_runner.cc:186] Error getting master registration for 127.6.68.253:38441: OK, instance_id { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" instance_seqno: 1714724202684737 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 08:16:42.886813 19815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 08:16:42.886703 20033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } }
I20240503 08:16:42.886855 20033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:42.886883 20033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070893ff3ab84ece8a300f3ef66a5f97, State: Initialized, Role: FOLLOWER
I20240503 08:16:42.887001 20033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } }
I20240503 08:16:42.887302 20033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:16:42.887279 20054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } } }
I20240503 08:16:42.887411 20054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:16:42.888909 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20240503 08:16:42.889196 20065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:16:42.889235 20065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:16:42.891476 19965 master_runner.cc:186] Error getting master registration for 127.6.68.253:38441: OK, instance_id { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" instance_seqno: 1714724202684737 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 08:16:42.894100 19965 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 08:16:42.898037 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.890200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.6.68.254:40729)
0503 08:16:42.890243 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.890245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.890403 (+ 158us) server_negotiation.cc:407] Connection header received
0503 08:16:42.890503 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.890505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.890515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.890544 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.891575 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.892290 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.893029 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.893210 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.894776 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.894781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.894781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.894790 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.896803 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.897174 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.897175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.897175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.897199 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.897397 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.897398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.897399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.897513 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.897599 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.897818 (+ 219us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.897913 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:16:42.905586 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.890639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.6.68.252:43905)
0503 08:16:42.890917 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.890919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.897196 (+ 6277us) server_negotiation.cc:407] Connection header received
0503 08:16:42.897298 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.897299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.897311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.897339 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.899739 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.900422 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.901161 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.901340 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.902657 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.902662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.902662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.902669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.904424 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.904824 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.904825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.904825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.904849 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.905000 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.905000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.905001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.905115 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.905185 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.905390 (+ 205us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.905477 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20240503 08:16:42.905864 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.890524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.6.68.253:38441)
0503 08:16:42.890615 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.890617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.895803 (+ 5186us) server_negotiation.cc:407] Connection header received
0503 08:16:42.896237 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.896239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.896251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.896283 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.898199 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.898873 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.900023 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.900269 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.902237 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.902242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.902242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.902250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.904654 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.905097 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.905098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.905098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.905134 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.905302 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.905303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.905303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.905414 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.905480 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.905684 (+ 204us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.905774 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20240503 08:16:42.920200 19958 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } attempt: 3
May 03 08:16:42 dist-test-slave-3g67 krb5kdc[19802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724202, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
I20240503 08:16:42.928022 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.920643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.6.68.252:43905)
0503 08:16:42.920707 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:42.920710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:42.920720 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:16:42.920749 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:42.920750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:42.920761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:42.920851 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:42.921444 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.922160 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.922921 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.923525 (+ 604us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.925119 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:42.925123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:42.925124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:42.925132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:42.926914 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.927299 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.927300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.927300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.927323 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:42.927473 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:42.927473 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:42.927474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:42.927596 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:42.927686 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:42.927835 (+ 149us) server_negotiation.cc:299] Negotiation successful
0503 08:16:42.927921 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240503 08:16:42.928023 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:42.920554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43905 (local address 127.0.0.1:41122)
0503 08:16:42.920614 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:42.920626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:42.920666 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:42.920922 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:42.920924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:42.921135 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:42.921342 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.921345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.922255 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:42.922256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:42.922832 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:42.922833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:42.922915 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:42.923480 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:42.923486 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:42.925018 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:42.927011 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.927012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.927013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.927219 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.927386 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:42.927387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:42.927387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:42.927418 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:42.927687 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:42.927688 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:42.927739 (+ 51us) client_negotiation.cc:769] Sending connection context
0503 08:16:42.927804 (+ 65us) client_negotiation.cc:240] Negotiation successful
0503 08:16:42.927910 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20240503 08:16:42.930022 19958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227: Bootstrap starting.
I20240503 08:16:42.930737 19958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:42.931020 19958 log.cc:824] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:42.931898 19958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227: No bootstrap required, opened a new log
I20240503 08:16:42.933841 19958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } }
I20240503 08:16:42.933964 19958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:42.933987 19958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a043ef94d94534aa0cdd0ba44b2227, State: Initialized, Role: FOLLOWER
I20240503 08:16:42.934099 19958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } }
I20240503 08:16:42.934432 19958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:16:42.934403 20075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } } }
I20240503 08:16:42.934495 20075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 08:16:42.936592 20086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:16:42.936708 20086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:16:42.938766 19890 master_runner.cc:263] No leader master found from master 83a043ef94d94534aa0cdd0ba44b2227
I20240503 08:16:42.938817 19890 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 08:16:42.979225 20040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:16:42.979357 20040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } }
I20240503 08:16:42.979605 20040 leader_election.cc:288] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83a043ef94d94534aa0cdd0ba44b2227 (127.6.68.253:38441), 070893ff3ab84ece8a300f3ef66a5f97 (127.6.68.252:43905)
I20240503 08:16:42.980299 19932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32daed9e1bb248888a7d16e41e963472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a043ef94d94534aa0cdd0ba44b2227" is_pre_election: true
I20240503 08:16:42.980458 19932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32daed9e1bb248888a7d16e41e963472 in term 0.
I20240503 08:16:42.980631 20007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32daed9e1bb248888a7d16e41e963472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070893ff3ab84ece8a300f3ef66a5f97" is_pre_election: true
I20240503 08:16:42.980759 20007 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32daed9e1bb248888a7d16e41e963472 in term 0.
I20240503 08:16:42.981194 19832 leader_election.cc:302] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32daed9e1bb248888a7d16e41e963472, 83a043ef94d94534aa0cdd0ba44b2227; no voters:
I20240503 08:16:42.981448 20040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:16:42.981510 20040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:16:42.981532 20040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:42.982571 20040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } }
I20240503 08:16:42.982707 20040 leader_election.cc:288] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [CANDIDATE]: Term 1 election: Requested vote from peers 83a043ef94d94534aa0cdd0ba44b2227 (127.6.68.253:38441), 070893ff3ab84ece8a300f3ef66a5f97 (127.6.68.252:43905)
I20240503 08:16:42.983098 20007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32daed9e1bb248888a7d16e41e963472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070893ff3ab84ece8a300f3ef66a5f97"
I20240503 08:16:42.983193 20007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:42.983762 19932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32daed9e1bb248888a7d16e41e963472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a043ef94d94534aa0cdd0ba44b2227"
I20240503 08:16:42.983850 19932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:42.984768 19932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32daed9e1bb248888a7d16e41e963472 in term 1.
I20240503 08:16:42.985399 20007 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32daed9e1bb248888a7d16e41e963472 in term 1.
I20240503 08:16:42.985889 19832 leader_election.cc:302] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 070893ff3ab84ece8a300f3ef66a5f97, 32daed9e1bb248888a7d16e41e963472; no voters:
I20240503 08:16:42.987105 20040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:16:42.987310 20040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [term 1 LEADER]: Becoming Leader. State: Replica: 32daed9e1bb248888a7d16e41e963472, State: Running, Role: LEADER
I20240503 08:16:42.987406 20040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } }
I20240503 08:16:42.987794 20040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32daed9e1bb248888a7d16e41e963472. Latest consensus state: current_term: 1 leader_uuid: "32daed9e1bb248888a7d16e41e963472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } } }
I20240503 08:16:42.987867 20040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:42.988978 20090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:16:42.989207 20090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:16:42.990875 20007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [term 1 FOLLOWER]: Refusing update from remote peer 32daed9e1bb248888a7d16e41e963472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:16:42.991283 20088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:16:42.992640 20054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32daed9e1bb248888a7d16e41e963472. Latest consensus state: current_term: 1 leader_uuid: "32daed9e1bb248888a7d16e41e963472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } } }
I20240503 08:16:42.992733 20054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:16:42.993232 19932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [term 1 FOLLOWER]: Refusing update from remote peer 32daed9e1bb248888a7d16e41e963472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:16:42.993559 20088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:16:42.994771 20075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32daed9e1bb248888a7d16e41e963472. Latest consensus state: current_term: 1 leader_uuid: "32daed9e1bb248888a7d16e41e963472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } } }
I20240503 08:16:42.994863 20075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:16:42.996089 20088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32daed9e1bb248888a7d16e41e963472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } } }
I20240503 08:16:42.996172 20088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:42.996629 20075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32daed9e1bb248888a7d16e41e963472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } } }
I20240503 08:16:42.996711 20075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:16:42.996958 20091 mvcc.cc:205] Tried to move back new op lower bound from 7023510335447900160 to 7023510335436898304. Current Snapshot: MvccSnapshot[applied={T|T < 7023510335447900160}]
I20240503 08:16:42.997963 20094 mvcc.cc:205] Tried to move back new op lower bound from 7023510335447900160 to 7023510335436898304. Current Snapshot: MvccSnapshot[applied={T|T < 7023510335447900160}]
I20240503 08:16:42.998338 20040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32daed9e1bb248888a7d16e41e963472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } } }
I20240503 08:16:42.998415 20040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:42.998824 20054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32daed9e1bb248888a7d16e41e963472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32daed9e1bb248888a7d16e41e963472" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40729 } } peers { permanent_uuid: "83a043ef94d94534aa0cdd0ba44b2227" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 38441 } } peers { permanent_uuid: "070893ff3ab84ece8a300f3ef66a5f97" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43905 } } }
I20240503 08:16:42.998914 20054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:16:42.999557 20092 mvcc.cc:205] Tried to move back new op lower bound from 7023510335447900160 to 7023510335436898304. Current Snapshot: MvccSnapshot[applied={T|T < 7023510335447900160}]
I20240503 08:16:43.001259 20090 catalog_manager.cc:1347] Generated new cluster ID: e178c8e1d026454faf07272e1ad76bee
I20240503 08:16:43.001319 20090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:16:43.013953 20090 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:16:43.014868 20090 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:16:43.026968 20090 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 32daed9e1bb248888a7d16e41e963472: Generated new TSK 0
I20240503 08:16:43.027323 20090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:16:43.889662 20065 catalog_manager.cc:1259] Loaded cluster ID: e178c8e1d026454faf07272e1ad76bee
I20240503 08:16:43.889744 20065 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97: loading cluster ID for follower catalog manager: success
I20240503 08:16:43.890722 20065 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97: acquiring CA information for follower catalog manager: success
I20240503 08:16:43.891021 20065 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 070893ff3ab84ece8a300f3ef66a5f97: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:16:43.937250 20086 catalog_manager.cc:1259] Loaded cluster ID: e178c8e1d026454faf07272e1ad76bee
I20240503 08:16:43.937342 20086 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227: loading cluster ID for follower catalog manager: success
I20240503 08:16:43.938287 20086 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227: acquiring CA information for follower catalog manager: success
I20240503 08:16:43.938643 20086 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 83a043ef94d94534aa0cdd0ba44b2227: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:16:44.081972 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 19815
I20240503 08:16:44.087601 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 19890
I20240503 08:16:44.092655 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 19965
2024-05-03T08:16:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2781 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:16:44 dist-test-slave-3g67 krb5kdc[20109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:16:44 dist-test-slave-3g67 krb5kdc[20109](info): set up 2 sockets
May 03 08:16:44 dist-test-slave-3g67 krb5kdc[20109](info): commencing operation
krb5kdc: starting...
W20240503 08:16:45.059981 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:16:45 dist-test-slave-3g67 krb5kdc[20109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:16:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:16:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:16:45.178619 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:32825
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:32825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:45.259104 20122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:45.259326 20122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:45.259349 20122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:45.260694 20122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:16:45.260738 20122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:45.260753 20122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:45.260766 20122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:16:45.260776 20122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:16:45.262352 20122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:32825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:45.262538 20122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:45.262732 20122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:45.267217 20128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:45.267813 20130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:45.268420 20127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:45.269235 20122 server_base.cc:1034] running on GCE node
I20240503 08:16:45.269428 20122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:45.269639 20122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:45.270784 20122 hybrid_clock.cc:648] HybridClock initialized: now 1714724205270764 us; error 38 us; skew 500 ppm
May 03 08:16:45 dist-test-slave-3g67 krb5kdc[20109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724205, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:45.277366 20122 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:16:45.277796 20122 webserver.cc:460] Webserver started at http://127.6.68.254:35335/ using document root <none> and password file <none>
I20240503 08:16:45.277980 20122 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:45.278023 20122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:45.278129 20122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:16:45.279120 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "6bb5b4039e384248b619fb9b368ff1e7"
format_stamp: "Formatted at 2024-05-03 08:16:45 on dist-test-slave-3g67"
server_key: "e83c9d54d2bc2d463ca1c1dfffdae957"
server_key_iv: "0466ba46e1698df4745c8d2749bd10c3"
server_key_version: "encryptionkey@0"
I20240503 08:16:45.279434 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "6bb5b4039e384248b619fb9b368ff1e7"
format_stamp: "Formatted at 2024-05-03 08:16:45 on dist-test-slave-3g67"
server_key: "e83c9d54d2bc2d463ca1c1dfffdae957"
server_key_iv: "0466ba46e1698df4745c8d2749bd10c3"
server_key_version: "encryptionkey@0"
I20240503 08:16:45.280733 20122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:16:45.281428 20137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:45.281595 20122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:16:45.281673 20122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "6bb5b4039e384248b619fb9b368ff1e7"
format_stamp: "Formatted at 2024-05-03 08:16:45 on dist-test-slave-3g67"
server_key: "e83c9d54d2bc2d463ca1c1dfffdae957"
server_key_iv: "0466ba46e1698df4745c8d2749bd10c3"
server_key_version: "encryptionkey@0"
I20240503 08:16:45.281738 20122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:45.296118 20122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:45.298278 20122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:45.298413 20122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:45.302297 20122 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:32825
I20240503 08:16:45.302330 20189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:32825 every 8 connection(s)
I20240503 08:16:45.302645 20122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:16:45.303303 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 20122
I20240503 08:16:45.303385 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:16:45.303450 20190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:45.303550 6419 external_mini_cluster.cc:1428] Setting key c216b77ef896076c168bebf5d5f0c37d
I20240503 08:16:45.306555 20190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7: Bootstrap starting.
I20240503 08:16:45.307179 20190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:45.307432 20190 log.cc:824] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:45.308215 20190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7: No bootstrap required, opened a new log
May 03 08:16:45 dist-test-slave-3g67 krb5kdc[20109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:45.310293 20190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb5b4039e384248b619fb9b368ff1e7" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 32825 } }
I20240503 08:16:45.310421 20190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:45.310454 20190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb5b4039e384248b619fb9b368ff1e7, State: Initialized, Role: FOLLOWER
I20240503 08:16:45.310575 20190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb5b4039e384248b619fb9b368ff1e7" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 32825 } }
I20240503 08:16:45.310657 20190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:16:45.310700 20190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:16:45.310748 20190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:45.311470 20190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb5b4039e384248b619fb9b368ff1e7" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 32825 } }
I20240503 08:16:45.311558 20190 leader_election.cc:302] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bb5b4039e384248b619fb9b368ff1e7; no voters:
I20240503 08:16:45.311749 20190 leader_election.cc:288] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:16:45.311808 20195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:16:45.311980 20190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:16:45.311986 20195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [term 1 LEADER]: Becoming Leader. State: Replica: 6bb5b4039e384248b619fb9b368ff1e7, State: Running, Role: LEADER
I20240503 08:16:45.312099 20195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb5b4039e384248b619fb9b368ff1e7" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 32825 } }
I20240503 08:16:45.312892 20197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bb5b4039e384248b619fb9b368ff1e7. Latest consensus state: current_term: 1 leader_uuid: "6bb5b4039e384248b619fb9b368ff1e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb5b4039e384248b619fb9b368ff1e7" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 32825 } } }
I20240503 08:16:45.312974 20197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:45.313041 20196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bb5b4039e384248b619fb9b368ff1e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb5b4039e384248b619fb9b368ff1e7" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 32825 } } }
I20240503 08:16:45.313094 20196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7 [sys.catalog]: This master's current role is: LEADER
W20240503 08:16:45.313695 20210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:16:45.313776 20210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:16:45.313822 20206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:16:45.313987 20206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:16:45.314239 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:45.304104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.6.68.254:32825)
0503 08:16:45.304283 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:45.304295 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:45.304319 (+ 24us) server_negotiation.cc:407] Connection header received
0503 08:16:45.304965 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:45.304974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:45.305032 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:45.305191 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:45.305931 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:45.307037 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:45.307661 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:45.307837 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:45.310208 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:45.310225 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:45.310225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:45.310242 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:45.312268 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:45.312671 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:45.312674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:45.312675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:45.312755 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:45.312944 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:45.312945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:45.312945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:45.313242 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:45.313407 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:45.313967 (+ 560us) server_negotiation.cc:299] Negotiation successful
0503 08:16:45.314090 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20240503 08:16:45.315172 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:16:45.315416 20206 catalog_manager.cc:1347] Generated new cluster ID: 118e0e7698cb4977abf122241ac8df62
I20240503 08:16:45.315464 20206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:16:45.320143 20206 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:16:45.320861 20206 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:16:45.327528 20206 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6bb5b4039e384248b619fb9b368ff1e7: Generated new TSK 0
I20240503 08:16:45.327741 20206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:16:45.368851 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 20122
2024-05-03T08:16:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1274 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:16:45 dist-test-slave-3g67 krb5kdc[20217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:16:45 dist-test-slave-3g67 krb5kdc[20217](info): set up 2 sockets
May 03 08:16:45 dist-test-slave-3g67 krb5kdc[20217](info): commencing operation
krb5kdc: starting...
W20240503 08:16:46.316061 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:16:46 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:16:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:16:46.434861 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:42893
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:42893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:46.513819 20230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:46.514022 20230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:46.514045 20230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:46.515331 20230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:16:46.515367 20230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:46.515380 20230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:46.515393 20230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:16:46.515404 20230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:16:46.516963 20230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:42893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:42893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:46.517143 20230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:46.517297 20230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:46.521970 20238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:46.522549 20236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:46.523126 20235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:46.523782 20230 server_base.cc:1034] running on GCE node
I20240503 08:16:46.524000 20230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:46.524247 20230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:46.525408 20230 hybrid_clock.cc:648] HybridClock initialized: now 1714724206525385 us; error 33 us; skew 500 ppm
May 03 08:16:46 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724206, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:46.527956 20230 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:16:46.528379 20230 webserver.cc:460] Webserver started at http://127.6.68.254:44031/ using document root <none> and password file <none>
I20240503 08:16:46.528553 20230 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:46.528589 20230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:46.528677 20230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:16:46.529645 20230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "32a73b315657428d8b198a48d7766b89"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "c2b9766d801f054978b445d5c37c9ffe"
server_key_iv: "5129250375a07c9d6787a9fc6c9ef839"
server_key_version: "encryptionkey@0"
I20240503 08:16:46.529908 20230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "32a73b315657428d8b198a48d7766b89"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "c2b9766d801f054978b445d5c37c9ffe"
server_key_iv: "5129250375a07c9d6787a9fc6c9ef839"
server_key_version: "encryptionkey@0"
I20240503 08:16:46.531059 20230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:16:46.531720 20245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:46.531891 20230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:46.531967 20230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "32a73b315657428d8b198a48d7766b89"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "c2b9766d801f054978b445d5c37c9ffe"
server_key_iv: "5129250375a07c9d6787a9fc6c9ef839"
server_key_version: "encryptionkey@0"
I20240503 08:16:46.532033 20230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:46.551304 20230 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:46.553098 20230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:46.553227 20230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:46.556933 20230 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:42893
I20240503 08:16:46.556983 20297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:42893 every 8 connection(s)
I20240503 08:16:46.557293 20230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:16:46.558087 20298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:46.559901 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 20230
I20240503 08:16:46.560005 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:16:46.560158 6419 external_mini_cluster.cc:1428] Setting key e8935c47aa352f63529e6fffe956b5d4
I20240503 08:16:46.560261 20298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Bootstrap starting.
I20240503 08:16:46.560956 20298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:46.561244 20298 log.cc:824] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:46.561949 20298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: No bootstrap required, opened a new log
I20240503 08:16:46.563869 20298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:46.564008 20298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:46.564028 20298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a73b315657428d8b198a48d7766b89, State: Initialized, Role: FOLLOWER
I20240503 08:16:46.564121 20298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:46.564185 20298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:16:46.564210 20298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:16:46.564257 20298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:46.564934 20298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:46.565022 20298 leader_election.cc:302] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32a73b315657428d8b198a48d7766b89; no voters:
I20240503 08:16:46.565176 20298 leader_election.cc:288] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:16:46.565227 20303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:16:46.565403 20303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 1 LEADER]: Becoming Leader. State: Replica: 32a73b315657428d8b198a48d7766b89, State: Running, Role: LEADER
I20240503 08:16:46.565418 20298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: configured and running, proceeding with master startup.
May 03 08:16:46 dist-test-slave-3g67 krb5kdc[20217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:46.566826 20303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:46.567416 20305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32a73b315657428d8b198a48d7766b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } } }
I20240503 08:16:46.569180 20305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:46.567786 20304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32a73b315657428d8b198a48d7766b89. Latest consensus state: current_term: 1 leader_uuid: "32a73b315657428d8b198a48d7766b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } } }
W20240503 08:16:46.567893 20317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:16:46.569391 20317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:16:46.569463 20304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:46.569586 20320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:16:46.569761 20320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:16:46.570524 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:46.560740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.6.68.254:42893)
0503 08:16:46.560967 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:46.560979 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:46.561004 (+ 25us) server_negotiation.cc:407] Connection header received
0503 08:16:46.561583 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:46.561590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:46.561638 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:46.561790 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:46.562518 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:46.563814 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:46.564488 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:46.564723 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:46.567106 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:46.567115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:46.567115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:46.567135 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:46.568964 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:46.569323 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:46.569326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:46.569327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:46.569386 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:46.569570 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:46.569571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:46.569571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:46.569792 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:46.569947 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:46.570206 (+ 259us) server_negotiation.cc:299] Negotiation successful
0503 08:16:46.570346 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":117,"threads_started":1}
I20240503 08:16:46.571195 20320 catalog_manager.cc:1347] Generated new cluster ID: d008473838e547b39221ab937cbf4439
I20240503 08:16:46.571251 20320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:16:46.587097 20320 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:16:46.587613 20320 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:16:46.598330 20320 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Generated new TSK 0
I20240503 08:16:46.598497 20320 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:16:46.627135 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:42893
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:46.704759 20324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:46.704943 20324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:46.704967 20324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:46.706245 20324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:46.706295 20324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:46.706321 20324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:16:46.707902 20324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:46.708096 20324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:46.708292 20324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:16:46.713330 20324 server_base.cc:1034] running on GCE node
W20240503 08:16:46.713908 20332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:46.714524 20329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:46.715051 20330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:46.715270 20324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:46.715452 20324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:46.716624 20324 hybrid_clock.cc:648] HybridClock initialized: now 1714724206716600 us; error 40 us; skew 500 ppm
May 03 08:16:46 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724206, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:46.718945 20324 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:16:46.719347 20324 webserver.cc:460] Webserver started at http://127.6.68.193:33191/ using document root <none> and password file <none>
I20240503 08:16:46.719516 20324 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:46.719553 20324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:46.719708 20324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:16:46.720702 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "23145c47f85941baaf71efc312edbff2"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "a8b3ae597c8337a27f13aecfee710e57"
server_key_iv: "5c86af9eae0ba98f095282cdefde528d"
server_key_version: "encryptionkey@0"
I20240503 08:16:46.721010 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "23145c47f85941baaf71efc312edbff2"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "a8b3ae597c8337a27f13aecfee710e57"
server_key_iv: "5c86af9eae0ba98f095282cdefde528d"
server_key_version: "encryptionkey@0"
I20240503 08:16:46.722153 20324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:16:46.722798 20339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:46.722949 20324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:16:46.723012 20324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "23145c47f85941baaf71efc312edbff2"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "a8b3ae597c8337a27f13aecfee710e57"
server_key_iv: "5c86af9eae0ba98f095282cdefde528d"
server_key_version: "encryptionkey@0"
I20240503 08:16:46.723062 20324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:46.744805 20324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:46.746624 20324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:46.746728 20324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:46.746912 20324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:46.747159 20324 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:46.747186 20324 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:46.747207 20324 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:46.747232 20324 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:46.752651 20324 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:38923
I20240503 08:16:46.752704 20452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:38923 every 8 connection(s)
I20240503 08:16:46.752982 20324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
May 03 08:16:46 dist-test-slave-3g67 krb5kdc[20217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724206, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:46.760896 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:46.753676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:35511 (local address 127.6.68.254:42893)
0503 08:16:46.753771 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:46.753774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:46.753862 (+ 88us) server_negotiation.cc:407] Connection header received
0503 08:16:46.754326 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:46.754328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:46.754337 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:46.754390 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:46.755379 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:46.755918 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:46.756443 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:46.756555 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:46.758446 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:46.758450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:46.758450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:46.758456 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:46.759759 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:46.760167 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:46.760168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:46.760168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:46.760193 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:46.760372 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:46.760373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:46.760374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:46.760468 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:46.760523 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:46.760739 (+ 216us) server_negotiation.cc:299] Negotiation successful
0503 08:16:46.760815 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20240503 08:16:46.760980 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:46.753621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:35511)
0503 08:16:46.753761 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:46.753787 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:46.754225 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:46.754446 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:46.754450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:46.754734 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:46.755286 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:46.755290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:46.755988 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:46.755989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:46.756373 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:46.756375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:46.756475 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:46.756977 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:46.756993 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:46.758351 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:46.759849 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:46.759854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:46.759857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:46.760052 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:46.760255 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:46.760256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:46.760256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:46.760307 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:46.760524 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:46.760528 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:46.760632 (+ 104us) client_negotiation.cc:769] Sending connection context
0503 08:16:46.760717 (+ 85us) client_negotiation.cc:240] Negotiation successful
0503 08:16:46.760836 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20240503 08:16:46.761505 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 20324
I20240503 08:16:46.761547 20453 heartbeater.cc:347] Connected to a master server at 127.6.68.254:42893
I20240503 08:16:46.761624 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:16:46.761639 20453 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:46.761812 6419 external_mini_cluster.cc:1428] Setting key 8299847356a91d88553984e5c45b247d
I20240503 08:16:46.761883 20453 heartbeater.cc:511] Master 127.6.68.254:42893 requested a full tablet report, sending...
I20240503 08:16:46.762421 20261 ts_manager.cc:194] Registered new tserver with Master: 23145c47f85941baaf71efc312edbff2 (127.6.68.193:38923)
I20240503 08:16:46.763420 20261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:35511
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:16:46.787006 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:42893
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:46.865092 20458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:46.865299 20458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:46.865324 20458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:46.866879 20458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:46.866928 20458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:46.866955 20458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:16:46.868643 20458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:46.868847 20458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:46.869041 20458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:46.873893 20463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:46.874305 20466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:46.874806 20464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:46.875476 20458 server_base.cc:1034] running on GCE node
I20240503 08:16:46.875684 20458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:46.875905 20458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:46.877033 20458 hybrid_clock.cc:648] HybridClock initialized: now 1714724206877015 us; error 39 us; skew 500 ppm
May 03 08:16:46 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724206, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:46.879393 20458 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:16:46.879841 20458 webserver.cc:460] Webserver started at http://127.6.68.194:41865/ using document root <none> and password file <none>
I20240503 08:16:46.880024 20458 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:46.880070 20458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:46.880177 20458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:16:46.881189 20458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "75cd69107d6542aa94e6e1f29c43e4bb"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "085dac7bd783a7ed1a1a622bb1d8de94"
server_key_iv: "f5d3bee983c67eab029eb814bb7919f4"
server_key_version: "encryptionkey@0"
I20240503 08:16:46.881495 20458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "75cd69107d6542aa94e6e1f29c43e4bb"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "085dac7bd783a7ed1a1a622bb1d8de94"
server_key_iv: "f5d3bee983c67eab029eb814bb7919f4"
server_key_version: "encryptionkey@0"
I20240503 08:16:46.882725 20458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:16:46.883360 20473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:46.883527 20458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:16:46.883607 20458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "75cd69107d6542aa94e6e1f29c43e4bb"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "085dac7bd783a7ed1a1a622bb1d8de94"
server_key_iv: "f5d3bee983c67eab029eb814bb7919f4"
server_key_version: "encryptionkey@0"
I20240503 08:16:46.883709 20458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:46.898177 20458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:46.900226 20458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:46.900352 20458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:46.900557 20458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:46.900873 20458 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:46.900907 20458 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:46.900943 20458 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:46.900969 20458 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:46.906399 20458 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:38643
I20240503 08:16:46.906455 20586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:38643 every 8 connection(s)
I20240503 08:16:46.906745 20458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:16:46.911500 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 20458
I20240503 08:16:46.911587 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:16:46.911792 6419 external_mini_cluster.cc:1428] Setting key 22778651fda98dc7303048019bf2f4be
May 03 08:16:46 dist-test-slave-3g67 krb5kdc[20217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724206, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:46.915450 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:46.907488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:55191 (local address 127.6.68.254:42893)
0503 08:16:46.907558 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:46.907561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:46.907710 (+ 149us) server_negotiation.cc:407] Connection header received
0503 08:16:46.908210 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:46.908211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:46.908221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:46.908283 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:46.909268 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:46.909869 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:46.910424 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:46.910538 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:46.912585 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:46.912588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:46.912588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:46.912594 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:46.914202 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:46.914698 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:46.914699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:46.914699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:46.914716 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:46.914898 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:46.914898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:46.914898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:46.914973 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:46.915018 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:46.915312 (+ 294us) server_negotiation.cc:299] Negotiation successful
0503 08:16:46.915382 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240503 08:16:46.915701 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:46.907450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:55191)
0503 08:16:46.907594 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:46.907638 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:46.908098 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:46.908336 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:46.908340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:46.908633 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:46.909175 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:46.909179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:46.909967 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:46.909968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:46.910347 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:46.910348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:46.910462 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:46.910966 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:46.910982 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:46.912504 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:46.914305 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:46.914311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:46.914315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:46.914626 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:46.914768 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:46.914769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:46.914769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:46.914846 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:46.915022 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:46.915025 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:46.915165 (+ 140us) client_negotiation.cc:769] Sending connection context
0503 08:16:46.915285 (+ 120us) client_negotiation.cc:240] Negotiation successful
0503 08:16:46.915436 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20240503 08:16:46.916281 20587 heartbeater.cc:347] Connected to a master server at 127.6.68.254:42893
I20240503 08:16:46.916374 20587 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:46.916566 20587 heartbeater.cc:511] Master 127.6.68.254:42893 requested a full tablet report, sending...
I20240503 08:16:46.916941 20261 ts_manager.cc:194] Registered new tserver with Master: 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643)
I20240503 08:16:46.917305 20261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:55191
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:16:46.938396 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:42893
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:47.016233 20592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:47.016425 20592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:47.016456 20592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:47.017731 20592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:47.017776 20592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:47.017812 20592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:16:47.019326 20592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:47.019539 20592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:47.019748 20592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:47.024261 20597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:47.024876 20600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:47.025446 20598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:47.026032 20592 server_base.cc:1034] running on GCE node
I20240503 08:16:47.026186 20592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:47.026382 20592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:47.027524 20592 hybrid_clock.cc:648] HybridClock initialized: now 1714724207027492 us; error 42 us; skew 500 ppm
May 03 08:16:47 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:47.029979 20592 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:16:47.030398 20592 webserver.cc:460] Webserver started at http://127.6.68.195:40801/ using document root <none> and password file <none>
I20240503 08:16:47.030575 20592 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:47.030619 20592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:47.030714 20592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:16:47.031687 20592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "3a3385d0203b498d98801e167f26d07b"
format_stamp: "Formatted at 2024-05-03 08:16:47 on dist-test-slave-3g67"
server_key: "bcce4ee63d3c3def39ff10de1ae66aa8"
server_key_iv: "982d9806c32c51a9829890b2a1466c6d"
server_key_version: "encryptionkey@0"
I20240503 08:16:47.031986 20592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "3a3385d0203b498d98801e167f26d07b"
format_stamp: "Formatted at 2024-05-03 08:16:47 on dist-test-slave-3g67"
server_key: "bcce4ee63d3c3def39ff10de1ae66aa8"
server_key_iv: "982d9806c32c51a9829890b2a1466c6d"
server_key_version: "encryptionkey@0"
I20240503 08:16:47.033248 20592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:16:47.033952 20607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:47.034126 20592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:16:47.034204 20592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "3a3385d0203b498d98801e167f26d07b"
format_stamp: "Formatted at 2024-05-03 08:16:47 on dist-test-slave-3g67"
server_key: "bcce4ee63d3c3def39ff10de1ae66aa8"
server_key_iv: "982d9806c32c51a9829890b2a1466c6d"
server_key_version: "encryptionkey@0"
I20240503 08:16:47.034274 20592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:47.059037 20592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:47.060832 20592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:47.060952 20592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:47.061152 20592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:47.061451 20592 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:47.061483 20592 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:47.061518 20592 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:47.061555 20592 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:47.067219 20592 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:36181
I20240503 08:16:47.067274 20720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:36181 every 8 connection(s)
I20240503 08:16:47.067592 20592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:16:47.072917 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 20592
I20240503 08:16:47.073021 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
May 03 08:16:47 dist-test-slave-3g67 krb5kdc[20217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:47.073187 6419 external_mini_cluster.cc:1428] Setting key 96e464cc171617c513d53af430cc4082
I20240503 08:16:47.076186 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.068399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:36195 (local address 127.6.68.254:42893)
0503 08:16:47.068465 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.068467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.068624 (+ 157us) server_negotiation.cc:407] Connection header received
0503 08:16:47.069095 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.069096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.069109 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.069168 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:47.070183 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.070855 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.071449 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.071612 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.073699 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:47.073702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:47.073703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:47.073710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:47.075148 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.075493 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.075493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.075494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.075509 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.075666 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.075667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.075668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.075750 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:47.075807 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.076061 (+ 254us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.076124 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:16:47.076351 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.068357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:36195)
0503 08:16:47.068522 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.068546 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.068995 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.069237 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.069241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.069519 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:47.070081 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.070085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.070956 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.070957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.071362 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.071364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.071495 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.072044 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:47.072059 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:47.073609 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:47.075225 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.075229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.075232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.075433 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.075554 (+ 121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.075555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.075555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.075600 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.075832 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:47.075837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:47.075945 (+ 108us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.076041 (+ 96us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.076144 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20240503 08:16:47.076819 20721 heartbeater.cc:347] Connected to a master server at 127.6.68.254:42893
I20240503 08:16:47.076913 20721 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:47.077100 20721 heartbeater.cc:511] Master 127.6.68.254:42893 requested a full tablet report, sending...
I20240503 08:16:47.077481 20261 ts_manager.cc:194] Registered new tserver with Master: 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181)
I20240503 08:16:47.077844 20261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:36195
I20240503 08:16:47.085142 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:16:47.091603 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.086377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.6.68.254:42893)
0503 08:16:47.086420 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.086422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.086522 (+ 100us) server_negotiation.cc:407] Connection header received
0503 08:16:47.086548 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.086548 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.086559 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.086612 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:47.087266 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.087771 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.088289 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.088397 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.089335 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:47.089339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:47.089339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:47.089345 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:47.090750 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.091035 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.091036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.091036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.091077 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.091186 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.091187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.091187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.091287 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:47.091356 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.091426 (+ 70us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.091495 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:16:47.093505 20261 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:16:47.094880 20261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:16:47.100127 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.096824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:36181 (local address 127.0.0.1:59148)
0503 08:16:47.097107 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.097116 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.097168 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.097522 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.097522 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.097524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:47.097721 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.097723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.099138 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.099139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.099871 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.099873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.099940 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.099943 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.099975 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.100013 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":53,"threads_started":1}
I20240503 08:16:47.100035 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.097010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:38923 (local address 127.0.0.1:45184)
0503 08:16:47.097174 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.097184 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.097239 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.097479 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.097480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.097482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:47.097665 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.097667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.098613 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.098614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.099449 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.099451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.099526 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.099840 (+ 314us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.099897 (+ 57us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.099944 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":36,"threads_started":1}
I20240503 08:16:47.100737 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.096974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.6.68.194:38643)
0503 08:16:47.097143 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.097152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.097163 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:16:47.097208 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.097210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.097228 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.097305 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:47.097808 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.098977 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.100117 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.100584 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.100615 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.100641 (+ 26us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.100671 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20240503 08:16:47.100751 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.097177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45184 (local address 127.6.68.193:38923)
0503 08:16:47.097332 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.097339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.097351 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:16:47.097381 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.097383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.097395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.097468 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:47.097769 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.098531 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.099555 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.100116 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.100580 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.100619 (+ 39us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.100676 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20240503 08:16:47.100790 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.096925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.6.68.195:36181)
0503 08:16:47.097353 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.097360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.097370 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:16:47.097407 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.097409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.097424 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.097806 (+ 382us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:47.097886 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.099016 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.100084 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.100600 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.100639 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.100665 (+ 26us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.100703 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":92,"threads_started":1}
I20240503 08:16:47.100945 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.096787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38643 (local address 127.0.0.1:34212)
0503 08:16:47.096908 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.096933 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.096997 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.097406 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.097409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.097411 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:47.097702 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.097703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.099058 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.099058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.099929 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.099931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.099988 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.099991 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.100022 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.100057 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":77,"threads_started":1}
I20240503 08:16:47.101353 20655 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6d124145dc4171b5909b8c3f29399e (DEFAULT_TABLE table=test-table [id=49b7292fd8654eed933a936dbb438c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:16:47.101413 20521 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6d124145dc4171b5909b8c3f29399e (DEFAULT_TABLE table=test-table [id=49b7292fd8654eed933a936dbb438c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:16:47.101644 20521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6d124145dc4171b5909b8c3f29399e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:47.101655 20655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6d124145dc4171b5909b8c3f29399e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:47.104177 20736 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Bootstrap starting.
I20240503 08:16:47.104288 20735 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b: Bootstrap starting.
I20240503 08:16:47.104377 20387 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6d124145dc4171b5909b8c3f29399e (DEFAULT_TABLE table=test-table [id=49b7292fd8654eed933a936dbb438c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:16:47.104665 20387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6d124145dc4171b5909b8c3f29399e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:47.104900 20736 tablet_bootstrap.cc:654] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:47.105079 20735 tablet_bootstrap.cc:654] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:47.105195 20736 log.cc:824] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:47.105367 20735 log.cc:824] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:47.105863 20736 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: No bootstrap required, opened a new log
I20240503 08:16:47.105923 20736 ts_tablet_manager.cc:1367] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:16:47.106081 20735 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b: No bootstrap required, opened a new log
I20240503 08:16:47.106155 20735 ts_tablet_manager.cc:1367] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:16:47.107509 20736 raft_consensus.cc:348] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.107674 20736 raft_consensus.cc:374] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:47.107709 20736 raft_consensus.cc:729] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cd69107d6542aa94e6e1f29c43e4bb, State: Initialized, Role: FOLLOWER
I20240503 08:16:47.107662 20739 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Bootstrap starting.
I20240503 08:16:47.107820 20736 consensus_queue.cc:260] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.108044 20736 ts_tablet_manager.cc:1398] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:16:47.108166 20587 heartbeater.cc:503] Master 127.6.68.254:42893 was elected leader, sending a full tablet report...
I20240503 08:16:47.108364 20739 tablet_bootstrap.cc:654] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:47.108373 20735 raft_consensus.cc:348] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.108497 20735 raft_consensus.cc:374] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:47.108522 20735 raft_consensus.cc:729] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3385d0203b498d98801e167f26d07b, State: Initialized, Role: FOLLOWER
I20240503 08:16:47.108639 20739 log.cc:824] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:47.108608 20735 consensus_queue.cc:260] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.108844 20735 ts_tablet_manager.cc:1398] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:16:47.108884 20721 heartbeater.cc:503] Master 127.6.68.254:42893 was elected leader, sending a full tablet report...
I20240503 08:16:47.109297 20739 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: No bootstrap required, opened a new log
I20240503 08:16:47.109370 20739 ts_tablet_manager.cc:1367] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:16:47.110975 20739 raft_consensus.cc:348] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.111078 20739 raft_consensus.cc:374] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:47.111095 20739 raft_consensus.cc:729] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23145c47f85941baaf71efc312edbff2, State: Initialized, Role: FOLLOWER
I20240503 08:16:47.111173 20739 consensus_queue.cc:260] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.111368 20739 ts_tablet_manager.cc:1398] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:16:47.111425 20453 heartbeater.cc:503] Master 127.6.68.254:42893 was elected leader, sending a full tablet report...
W20240503 08:16:47.157272 20588 tablet.cc:2369] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:16:47.250492 20741 raft_consensus.cc:482] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:16:47.250624 20741 raft_consensus.cc:504] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.250967 20741 leader_election.cc:288] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643), 23145c47f85941baaf71efc312edbff2 (127.6.68.193:38923)
W20240503 08:16:47.253706 20454 tablet.cc:2369] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:16:47.253765 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.251149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38643 (local address 127.6.68.195:33235)
0503 08:16:47.251210 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.251222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.251299 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.251450 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.251451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.251453 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:47.251717 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.251719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.252605 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.252606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.253464 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.253465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.253539 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.253542 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.253598 (+ 56us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.253640 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20240503 08:16:47.253774 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.251273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:38923 (local address 127.6.68.195:52907)
0503 08:16:47.251442 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.251450 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.251506 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.251645 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.251646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.251647 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:47.251868 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.251869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.252709 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.252709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.253558 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.253560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.253618 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.253619 (+ 1us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.253649 (+ 30us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.253675 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":116,"threads_started":1}
I20240503 08:16:47.254134 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.251240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:33235 (local address 127.6.68.194:38643)
0503 08:16:47.251304 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.251307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.251321 (+ 14us) server_negotiation.cc:407] Connection header received
0503 08:16:47.251358 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.251359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.251371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.251442 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:47.251802 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.252522 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.253540 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.253989 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.253999 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.254022 (+ 23us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.254059 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:16:47.254410 20541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6d124145dc4171b5909b8c3f29399e" candidate_uuid: "3a3385d0203b498d98801e167f26d07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" is_pre_election: true
I20240503 08:16:47.254490 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.251381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:52907 (local address 127.6.68.193:38923)
0503 08:16:47.251424 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.251426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.251515 (+ 89us) server_negotiation.cc:407] Connection header received
0503 08:16:47.251557 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.251557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.251564 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.251614 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:47.251935 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.252623 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.253949 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.254394 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.254401 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.254419 (+ 18us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.254442 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:16:47.254524 20541 raft_consensus.cc:2454] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3385d0203b498d98801e167f26d07b in term 0.
I20240503 08:16:47.254734 20407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6d124145dc4171b5909b8c3f29399e" candidate_uuid: "3a3385d0203b498d98801e167f26d07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23145c47f85941baaf71efc312edbff2" is_pre_election: true
I20240503 08:16:47.254761 20608 leader_election.cc:302] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3385d0203b498d98801e167f26d07b, 75cd69107d6542aa94e6e1f29c43e4bb; no voters:
I20240503 08:16:47.254840 20407 raft_consensus.cc:2454] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3385d0203b498d98801e167f26d07b in term 0.
I20240503 08:16:47.254890 20741 raft_consensus.cc:2781] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:16:47.254967 20741 raft_consensus.cc:482] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:16:47.255002 20741 raft_consensus.cc:3037] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:47.255717 20741 raft_consensus.cc:504] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.255839 20741 leader_election.cc:288] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [CANDIDATE]: Term 1 election: Requested vote from peers 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643), 23145c47f85941baaf71efc312edbff2 (127.6.68.193:38923)
I20240503 08:16:47.256042 20407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6d124145dc4171b5909b8c3f29399e" candidate_uuid: "3a3385d0203b498d98801e167f26d07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23145c47f85941baaf71efc312edbff2"
I20240503 08:16:47.256045 20541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6d124145dc4171b5909b8c3f29399e" candidate_uuid: "3a3385d0203b498d98801e167f26d07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75cd69107d6542aa94e6e1f29c43e4bb"
I20240503 08:16:47.256107 20541 raft_consensus.cc:3037] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:47.256111 20407 raft_consensus.cc:3037] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:47.256738 20407 raft_consensus.cc:2454] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3385d0203b498d98801e167f26d07b in term 1.
I20240503 08:16:47.256760 20541 raft_consensus.cc:2454] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3385d0203b498d98801e167f26d07b in term 1.
I20240503 08:16:47.256912 20608 leader_election.cc:302] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23145c47f85941baaf71efc312edbff2, 3a3385d0203b498d98801e167f26d07b; no voters:
I20240503 08:16:47.257012 20741 raft_consensus.cc:2781] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:16:47.257185 20741 raft_consensus.cc:686] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 1 LEADER]: Becoming Leader. State: Replica: 3a3385d0203b498d98801e167f26d07b, State: Running, Role: LEADER
I20240503 08:16:47.257272 20741 consensus_queue.cc:237] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.258153 20261 catalog_manager.cc:5484] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3385d0203b498d98801e167f26d07b (127.6.68.195). New cstate: current_term: 1 leader_uuid: "3a3385d0203b498d98801e167f26d07b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } health_report { overall_health: UNKNOWN } } }
I20240503 08:16:47.281925 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.279793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.6.68.195:36181)
0503 08:16:47.279851 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.279853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.279860 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:16:47.279898 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.279898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.279914 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.279978 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:16:47.280333 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.280811 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.281389 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.281502 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.281533 (+ 31us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:16:47.281608 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:16:47.281665 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.281797 (+ 132us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.281832 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240503 08:16:47.283677 20407 raft_consensus.cc:1261] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Refusing update from remote peer 3a3385d0203b498d98801e167f26d07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:16:47.283676 20541 raft_consensus.cc:1261] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Refusing update from remote peer 3a3385d0203b498d98801e167f26d07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:16:47.284020 20745 consensus_queue.cc:1035] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:16:47.284130 20741 consensus_queue.cc:1035] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:16:47.286803 20750 mvcc.cc:205] Tried to move back new op lower bound from 7023510353031086080 to 7023510352926785536. Current Snapshot: MvccSnapshot[applied={T|T < 7023510353031086080}]
I20240503 08:16:47.286974 20749 mvcc.cc:205] Tried to move back new op lower bound from 7023510353031086080 to 7023510352926785536. Current Snapshot: MvccSnapshot[applied={T|T < 7023510353031086080}]
I20240503 08:16:47.288530 20262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41080:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:16:47.288626 20262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41080:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:16:47.289319 20262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Sending DeleteTablet for 3 replicas of tablet 1c6d124145dc4171b5909b8c3f29399e
I20240503 08:16:47.289589 20521 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6d124145dc4171b5909b8c3f29399e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:16:47 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:34212
I20240503 08:16:47.289636 20655 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6d124145dc4171b5909b8c3f29399e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:16:47 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:59148
I20240503 08:16:47.289654 20387 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6d124145dc4171b5909b8c3f29399e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:16:47 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:45184
I20240503 08:16:47.289860 20760 tablet_replica.cc:330] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: stopping tablet replica
I20240503 08:16:47.289891 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 20324
I20240503 08:16:47.289942 20760 raft_consensus.cc:2229] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:16:47.290867 20759 tablet_replica.cc:330] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b: stopping tablet replica
I20240503 08:16:47.290977 20759 raft_consensus.cc:2229] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:16:47.291111 20759 raft_consensus.cc:2258] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:16:47.291524 20759 ts_tablet_manager.cc:1875] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:16:47.293627 20759 ts_tablet_manager.cc:1888] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:16:47.293732 20759 log.cc:1196] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/1c6d124145dc4171b5909b8c3f29399e
I20240503 08:16:47.294174 20759 ts_tablet_manager.cc:1909] T 1c6d124145dc4171b5909b8c3f29399e P 3a3385d0203b498d98801e167f26d07b: Deleting consensus metadata
I20240503 08:16:47.294585 20249 catalog_manager.cc:4830] TS 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181): tablet 1c6d124145dc4171b5909b8c3f29399e (table test-table [id=49b7292fd8654eed933a936dbb438c5f]) successfully deleted
I20240503 08:16:47.295754 20758 tablet_replica.cc:330] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: stopping tablet replica
I20240503 08:16:47.295882 20758 raft_consensus.cc:2229] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:16:47.295991 20758 raft_consensus.cc:2258] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:16:47.296406 20758 ts_tablet_manager.cc:1875] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:16:47.296675 20246 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:38923 (error 108)
I20240503 08:16:47.296938 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 20458
I20240503 08:16:47.297582 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.297110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:38923 (local address 127.0.0.1:45200)
0503 08:16:47.297500 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.297542 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368}
W20240503 08:16:47.298259 20246 catalog_manager.cc:4573] TS 23145c47f85941baaf71efc312edbff2 (127.6.68.193:38923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c6d124145dc4171b5909b8c3f29399e: Network error: Client connection negotiation failed: client connection to 127.6.68.193:38923: connect: Connection refused (error 111)
I20240503 08:16:47.301913 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.301795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38643 (local address 127.0.0.1:34214)
0503 08:16:47.301851 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.301880 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.194:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20240503 08:16:47.301935 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 20592
I20240503 08:16:47.307142 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 20230
I20240503 08:16:47.312541 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:42893
--webserver_interface=127.6.68.254
--webserver_port=44031
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:42893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:47.389405 20762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:47.389585 20762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:47.389606 20762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:47.390810 20762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:16:47.390844 20762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:47.390858 20762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:47.390869 20762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:16:47.390882 20762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:16:47.392385 20762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:42893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:42893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:47.392544 20762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:47.392678 20762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:47.400416 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:47.400725 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:47.401458 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:47.402065 20762 server_base.cc:1034] running on GCE node
I20240503 08:16:47.402346 20762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:47.402572 20762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:47.403683 20762 hybrid_clock.cc:648] HybridClock initialized: now 1714724207403665 us; error 38 us; skew 500 ppm
May 03 08:16:47 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:47.406212 20762 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:16:47.406635 20762 webserver.cc:460] Webserver started at http://127.6.68.254:44031/ using document root <none> and password file <none>
I20240503 08:16:47.406824 20762 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:47.406872 20762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:47.408234 20762 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:47.408805 20778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:47.408974 20762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:16:47.409054 20762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "32a73b315657428d8b198a48d7766b89"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "c2b9766d801f054978b445d5c37c9ffe"
server_key_iv: "5129250375a07c9d6787a9fc6c9ef839"
server_key_version: "encryptionkey@0"
I20240503 08:16:47.409420 20762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:47.420396 20762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:47.422586 20762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:47.422717 20762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:47.426498 20762 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:42893
I20240503 08:16:47.426604 20830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:42893 every 8 connection(s)
I20240503 08:16:47.426879 20762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:16:47.428357 20831 sys_catalog.cc:263] Verifying existing consensus state
I20240503 08:16:47.428881 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Bootstrap starting.
I20240503 08:16:47.436729 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 20762
I20240503 08:16:47.437042 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:38923
--local_ip_for_outbound_sockets=127.6.68.193
--tserver_master_addrs=127.6.68.254:42893
--webserver_port=33191
--webserver_interface=127.6.68.193
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:16:47.437873 20831 log.cc:824] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:47.439764 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:16:47.440027 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Bootstrap complete.
I20240503 08:16:47.442153 20831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:47.442346 20831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a73b315657428d8b198a48d7766b89, State: Initialized, Role: FOLLOWER
I20240503 08:16:47.442468 20831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:47.442538 20831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:16:47.442574 20831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:16:47.442636 20831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:16:47.443503 20831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:47.443603 20831 leader_election.cc:302] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32a73b315657428d8b198a48d7766b89; no voters:
I20240503 08:16:47.443817 20831 leader_election.cc:288] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 08:16:47.443869 20835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:16:47.444028 20835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 2 LEADER]: Becoming Leader. State: Replica: 32a73b315657428d8b198a48d7766b89, State: Running, Role: LEADER
I20240503 08:16:47.444052 20831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:16:47.444183 20835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:47.444823 20837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32a73b315657428d8b198a48d7766b89. Latest consensus state: current_term: 2 leader_uuid: "32a73b315657428d8b198a48d7766b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } } }
I20240503 08:16:47.444903 20837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:47.445071 20845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:16:47.445114 20836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "32a73b315657428d8b198a48d7766b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } } }
I20240503 08:16:47.445226 20836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:47.445784 20845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:16:47.445922 20845 catalog_manager.cc:1259] Loaded cluster ID: d008473838e547b39221ab937cbf4439
I20240503 08:16:47.446156 20845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:16:47.446471 20845 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20240503 08:16:47.449380 20845 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20240503 08:16:47.451757 20845 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:16:47.451941 20845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Loaded TSK: 0
I20240503 08:16:47.452126 20845 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240503 08:16:47.516453 20833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:47.516636 20833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:47.516659 20833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:47.517935 20833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:47.517974 20833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:47.517999 20833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:16:47.519495 20833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:38923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=33191
--tserver_master_addrs=127.6.68.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:47.519701 20833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:47.519891 20833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:47.524366 20858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:47.524732 20861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:47.525472 20859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:47.526074 20833 server_base.cc:1034] running on GCE node
I20240503 08:16:47.526319 20833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:47.526568 20833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:47.527693 20833 hybrid_clock.cc:648] HybridClock initialized: now 1714724207527678 us; error 44 us; skew 500 ppm
May 03 08:16:47 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:47.534929 20833 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:16:47.535442 20833 webserver.cc:460] Webserver started at http://127.6.68.193:33191/ using document root <none> and password file <none>
I20240503 08:16:47.535689 20833 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:47.535768 20833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:47.537449 20833 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:16:47.538219 20868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:47.538405 20833 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:16:47.538472 20833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "23145c47f85941baaf71efc312edbff2"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "a8b3ae597c8337a27f13aecfee710e57"
server_key_iv: "5c86af9eae0ba98f095282cdefde528d"
server_key_version: "encryptionkey@0"
I20240503 08:16:47.538761 20833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:47.553035 20833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:47.555155 20833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:47.555253 20833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:47.555488 20833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:47.555925 20875 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 08:16:47.557076 20833 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:16:47.557122 20833 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:16:47.557159 20833 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:16:47.557689 20833 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:16:47.557721 20833 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 08:16:47.557797 20875 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Bootstrap starting.
I20240503 08:16:47.564985 20833 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:38923
I20240503 08:16:47.565068 20982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:38923 every 8 connection(s)
I20240503 08:16:47.565392 20833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:16:47.569866 20875 log.cc:824] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:47.572016 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 20833
I20240503 08:16:47.572291 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:38643
--local_ip_for_outbound_sockets=127.6.68.194
--tserver_master_addrs=127.6.68.254:42893
--webserver_port=41865
--webserver_interface=127.6.68.194
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 03 08:16:47 dist-test-slave-3g67 krb5kdc[20217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:47.574808 20875 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:16:47.575145 20875 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Bootstrap complete.
I20240503 08:16:47.576395 20875 ts_tablet_manager.cc:1367] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.015s
I20240503 08:16:47.576720 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.566043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:47975)
0503 08:16:47.567746 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.567796 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.568295 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.568667 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.568671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.569001 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:47.569288 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.569293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.570799 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.570800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.571181 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.571183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.571300 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.571998 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:47.572013 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:47.573457 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:47.575402 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.575407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.575410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.575601 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.575843 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.575844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.575844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.575900 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.576187 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:47.576192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:47.576315 (+ 123us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.576401 (+ 86us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.576540 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1658,"thread_start_us":81,"threads_started":1}
I20240503 08:16:47.576746 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.566128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:47975 (local address 127.6.68.254:42893)
0503 08:16:47.567758 (+ 1630us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.567770 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.567898 (+ 128us) server_negotiation.cc:407] Connection header received
0503 08:16:47.568490 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.568493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.568528 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.568638 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:47.569401 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.570712 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.571266 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.571485 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.573564 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:47.573589 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:47.573590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:47.573604 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:47.575320 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.575726 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.575731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.575731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.575780 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.575964 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.575965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.575966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.576123 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:47.576199 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.576479 (+ 280us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.576592 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1592,"thread_start_us":67,"threads_started":1}
I20240503 08:16:47.577410 20983 heartbeater.cc:347] Connected to a master server at 127.6.68.254:42893
I20240503 08:16:47.577497 20983 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:47.577682 20983 heartbeater.cc:511] Master 127.6.68.254:42893 requested a full tablet report, sending...
I20240503 08:16:47.578018 20875 raft_consensus.cc:348] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.578315 20875 raft_consensus.cc:729] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23145c47f85941baaf71efc312edbff2, State: Initialized, Role: FOLLOWER
I20240503 08:16:47.578316 20795 ts_manager.cc:194] Registered new tserver with Master: 23145c47f85941baaf71efc312edbff2 (127.6.68.193:38923)
I20240503 08:16:47.578462 20875 consensus_queue.cc:260] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.579061 20875 ts_tablet_manager.cc:1398] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:16:47.579427 20795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:47975
I20240503 08:16:47.580358 20983 heartbeater.cc:503] Master 127.6.68.254:42893 was elected leader, sending a full tablet report...
May 03 08:16:47 dist-test-slave-3g67 krb5kdc[20217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.193@KRBTEST.COM
I20240503 08:16:47.586745 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.578878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:38923 (local address 127.0.0.1:45208)
0503 08:16:47.579016 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.579038 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.579105 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.579973 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.579979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.580327 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:47.580611 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.580614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.581574 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.581575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.581961 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.581963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.582028 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.582598 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:47.582602 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:47.583842 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:47.585493 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.585499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.585501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.585729 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.585915 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.585916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.585916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.585951 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.586186 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:47.586191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:47.586265 (+ 74us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.586446 (+ 181us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.586608 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20240503 08:16:47.586751 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.579668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.6.68.193:38923)
0503 08:16:47.579811 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.579819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.579828 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:16:47.579854 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.579855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.579878 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.580686 (+ 808us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:47.580741 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.581497 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.582039 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.582280 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.583920 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:47.583931 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:47.583931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:47.583936 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:47.585423 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.585807 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.585810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.585810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.585841 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.585997 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.585998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.585999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.586102 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:47.586211 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.586543 (+ 332us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.586620 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20240503 08:16:47.587489 20917 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6d124145dc4171b5909b8c3f29399e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:16:47 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:45208
I20240503 08:16:47.588032 20992 tablet_replica.cc:330] stopping tablet replica
I20240503 08:16:47.588129 20992 raft_consensus.cc:2229] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:16:47.588214 20992 raft_consensus.cc:2258] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:16:47.588524 20992 ts_tablet_manager.cc:1875] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:16:47.590612 20992 ts_tablet_manager.cc:1888] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:16:47.590692 20992 log.cc:1196] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/1c6d124145dc4171b5909b8c3f29399e
I20240503 08:16:47.590940 20992 ts_tablet_manager.cc:1909] T 1c6d124145dc4171b5909b8c3f29399e P 23145c47f85941baaf71efc312edbff2: Deleting consensus metadata
I20240503 08:16:47.591411 20779 catalog_manager.cc:4830] TS 23145c47f85941baaf71efc312edbff2 (127.6.68.193:38923): tablet 1c6d124145dc4171b5909b8c3f29399e (table test-table [id=49b7292fd8654eed933a936dbb438c5f]) successfully deleted
W20240503 08:16:47.667662 20988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:47.667855 20988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:47.667877 20988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:47.669107 20988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:47.669143 20988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:47.669167 20988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:16:47.670646 20988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:38643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=41865
--tserver_master_addrs=127.6.68.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:47.670816 20988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:47.670958 20988 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:47.675807 21000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:47.676462 20997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:47.677109 20998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:47.677717 20988 server_base.cc:1034] running on GCE node
I20240503 08:16:47.678009 20988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:47.678231 20988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:47.679380 20988 hybrid_clock.cc:648] HybridClock initialized: now 1714724207679359 us; error 35 us; skew 500 ppm
May 03 08:16:47 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:47.681838 20988 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:16:47.682237 20988 webserver.cc:460] Webserver started at http://127.6.68.194:41865/ using document root <none> and password file <none>
I20240503 08:16:47.682433 20988 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:47.682492 20988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:47.683786 20988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:47.684474 21007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:47.684653 20988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:16:47.684726 20988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "75cd69107d6542aa94e6e1f29c43e4bb"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "085dac7bd783a7ed1a1a622bb1d8de94"
server_key_iv: "f5d3bee983c67eab029eb814bb7919f4"
server_key_version: "encryptionkey@0"
I20240503 08:16:47.684999 20988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:47.698925 20988 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:47.701047 20988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:47.701170 20988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:47.701382 20988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:47.701838 21014 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 08:16:47.702642 20988 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:16:47.702690 20988 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:16:47.702726 20988 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:16:47.703265 20988 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:16:47.703298 20988 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 08:16:47.703348 21014 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Bootstrap starting.
I20240503 08:16:47.704509 21014 log.cc:824] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:47.707741 21014 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:16:47.708125 21014 tablet_bootstrap.cc:492] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Bootstrap complete.
I20240503 08:16:47.708263 21014 ts_tablet_manager.cc:1367] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 08:16:47.709280 20988 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:38643
I20240503 08:16:47.709343 21122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:38643 every 8 connection(s)
I20240503 08:16:47.709618 20988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:16:47.709703 21014 raft_consensus.cc:348] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.710152 21014 raft_consensus.cc:729] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cd69107d6542aa94e6e1f29c43e4bb, State: Initialized, Role: FOLLOWER
I20240503 08:16:47.710290 21014 consensus_queue.cc:260] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.710546 21014 ts_tablet_manager.cc:1398] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:16:47.713246 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 20988
I20240503 08:16:47.713536 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:36181
--local_ip_for_outbound_sockets=127.6.68.195
--tserver_master_addrs=127.6.68.254:42893
--webserver_port=40801
--webserver_interface=127.6.68.195
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 03 08:16:47 dist-test-slave-3g67 krb5kdc[20217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:47.718233 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.710252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:43879 (local address 127.6.68.254:42893)
0503 08:16:47.710333 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.710336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.710548 (+ 212us) server_negotiation.cc:407] Connection header received
0503 08:16:47.710692 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.710694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.710706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.710768 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:47.711488 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.712043 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.712623 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.712798 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.715176 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:47.715180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:47.715180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:47.715186 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:47.716941 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.717416 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.717418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.717418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.717443 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.717660 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.717660 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.717661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.717741 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:47.717787 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.718097 (+ 310us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.718169 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240503 08:16:47.718544 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.710225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:43879)
0503 08:16:47.710428 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.710468 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.710603 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.710829 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.710832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.711135 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:47.711396 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.711400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.712125 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.712126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.712542 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.712543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.712644 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.713331 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:47.713346 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:47.715083 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:47.717042 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.717048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.717051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.717340 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.717518 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.717519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.717520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.717602 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.717800 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:47.717805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:47.717958 (+ 153us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.718091 (+ 133us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.718250 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20240503 08:16:47.719168 21123 heartbeater.cc:347] Connected to a master server at 127.6.68.254:42893
I20240503 08:16:47.719259 21123 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:47.719462 21123 heartbeater.cc:511] Master 127.6.68.254:42893 requested a full tablet report, sending...
I20240503 08:16:47.720115 20795 ts_manager.cc:194] Registered new tserver with Master: 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643)
I20240503 08:16:47.720739 20795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:43879
I20240503 08:16:47.722002 21123 heartbeater.cc:503] Master 127.6.68.254:42893 was elected leader, sending a full tablet report...
May 03 08:16:47 dist-test-slave-3g67 krb5kdc[20217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.194@KRBTEST.COM
I20240503 08:16:47.727362 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.720497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.6.68.194:38643)
0503 08:16:47.720659 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.720666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.720675 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:16:47.720704 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.720706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.720718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.720789 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:47.721304 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.722038 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.722894 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.723034 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.724748 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:47.724755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:47.724756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:47.724761 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:47.726406 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.726756 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.726759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.726759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.726795 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.726939 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.726940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.726940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.727030 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:47.727090 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.727223 (+ 133us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.727290 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20240503 08:16:47.727386 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.720421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38643 (local address 127.0.0.1:34230)
0503 08:16:47.720467 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.720476 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.720512 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.720812 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.720814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.721009 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:47.721218 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.721221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.722137 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.722138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.722699 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.722701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.722747 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.723230 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:47.723234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:47.724652 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:47.726489 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.726490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.726490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.726663 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.726856 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.726857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.726857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.726886 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.727079 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:47.727080 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:47.727130 (+ 50us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.727197 (+ 67us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.727294 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20240503 08:16:47.727794 21044 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6d124145dc4171b5909b8c3f29399e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:16:47 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:34230
I20240503 08:16:47.728406 21129 tablet_replica.cc:330] stopping tablet replica
I20240503 08:16:47.728494 21129 raft_consensus.cc:2229] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:16:47.728556 21129 raft_consensus.cc:2258] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:16:47.728834 21129 ts_tablet_manager.cc:1875] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:16:47.730787 21129 ts_tablet_manager.cc:1888] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:16:47.730850 21129 log.cc:1196] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/1c6d124145dc4171b5909b8c3f29399e
I20240503 08:16:47.731076 21129 ts_tablet_manager.cc:1909] T 1c6d124145dc4171b5909b8c3f29399e P 75cd69107d6542aa94e6e1f29c43e4bb: Deleting consensus metadata
I20240503 08:16:47.731492 20779 catalog_manager.cc:4830] TS 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643): tablet 1c6d124145dc4171b5909b8c3f29399e (table test-table [id=49b7292fd8654eed933a936dbb438c5f]) successfully deleted
W20240503 08:16:47.794636 21127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:47.794833 21127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:47.794857 21127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:47.796164 21127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:47.796209 21127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:47.796233 21127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:16:47.797762 21127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:36181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=40801
--tserver_master_addrs=127.6.68.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:47.797967 21127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:47.798156 21127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:47.802785 21134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:47.803490 21137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:47.804086 21135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:47.804682 21127 server_base.cc:1034] running on GCE node
I20240503 08:16:47.804906 21127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:47.805116 21127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:47.806272 21127 hybrid_clock.cc:648] HybridClock initialized: now 1714724207806253 us; error 38 us; skew 500 ppm
May 03 08:16:47 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:47.808728 21127 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:16:47.809115 21127 webserver.cc:460] Webserver started at http://127.6.68.195:40801/ using document root <none> and password file <none>
I20240503 08:16:47.809302 21127 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:47.809361 21127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:47.810603 21127 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:47.811167 21144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:47.811339 21127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:16:47.811414 21127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "3a3385d0203b498d98801e167f26d07b"
format_stamp: "Formatted at 2024-05-03 08:16:47 on dist-test-slave-3g67"
server_key: "bcce4ee63d3c3def39ff10de1ae66aa8"
server_key_iv: "982d9806c32c51a9829890b2a1466c6d"
server_key_version: "encryptionkey@0"
I20240503 08:16:47.811729 21127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:47.837316 21127 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:47.839355 21127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:47.839475 21127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:47.839717 21127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:47.840003 21127 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:47.840034 21127 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:47.840070 21127 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:47.840102 21127 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:47.845639 21127 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:36181
I20240503 08:16:47.845690 21257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:36181 every 8 connection(s)
I20240503 08:16:47.845988 21127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:16:47.848623 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 21127
May 03 08:16:47 dist-test-slave-3g67 krb5kdc[20217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:16:47.855401 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.849019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.6.68.254:42893)
0503 08:16:47.849131 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.849133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.849162 (+ 29us) server_negotiation.cc:407] Connection header received
0503 08:16:47.849236 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.849237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.849249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.849319 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:47.850138 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.850822 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.851594 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.851783 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.852783 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:47.852786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:47.852787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:47.852799 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:47.854387 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.854720 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.854721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.854721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.854745 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.854891 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.854892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.854892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.855037 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:47.855166 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.855240 (+ 74us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.855311 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":66,"threads_started":1}
I20240503 08:16:47.855712 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.846624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:60967)
0503 08:16:47.846747 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.846773 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.847235 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.847477 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.847481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.847831 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:47.848432 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.848438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.849409 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.849411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.850017 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.850019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.850170 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.850774 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:47.850790 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:47.852577 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:47.854364 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.854369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.854372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.854654 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.854808 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:47.854809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:47.854809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:47.854888 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:47.855101 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:47.855104 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:47.855265 (+ 161us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.855386 (+ 121us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.855529 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
I20240503 08:16:47.855571 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.846621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:60967 (local address 127.6.68.254:42893)
0503 08:16:47.846682 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.846684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.846851 (+ 167us) server_negotiation.cc:407] Connection header received
0503 08:16:47.847339 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.847340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.847355 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.847418 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:47.848562 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.849317 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.850097 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.850259 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.852675 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:47.852678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:47.852679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:47.852684 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:47.854276 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.854720 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.854721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.854721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.854745 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:47.854943 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:47.854944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:47.854944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:47.855037 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:47.855116 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.855409 (+ 293us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.855491 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240503 08:16:47.856225 21258 heartbeater.cc:347] Connected to a master server at 127.6.68.254:42893
I20240503 08:16:47.856323 21258 heartbeater.cc:464] Registering TS with master...
I20240503 08:16:47.856520 21258 heartbeater.cc:511] Master 127.6.68.254:42893 requested a full tablet report, sending...
I20240503 08:16:47.856895 20795 ts_manager.cc:194] Registered new tserver with Master: 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181)
I20240503 08:16:47.857272 20795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:60967
I20240503 08:16:47.868108 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:16:47.870746 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.868516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.6.68.254:42893)
0503 08:16:47.868567 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.868569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.868677 (+ 108us) server_negotiation.cc:407] Connection header received
0503 08:16:47.868764 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.868764 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.868788 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.868854 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:16:47.869192 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.869666 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.870245 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.870378 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.870418 (+ 40us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:16:47.870500 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:16:47.870576 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.870667 (+ 91us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.870690 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240503 08:16:47.871115 20795 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:16:47.872675 20795 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:16:47.874750 21044 tablet_service.cc:1467] Processing CreateTablet for tablet 578178d978164b868d6b894b0932325a (DEFAULT_TABLE table=test-table [id=50ab6dfb0f744cbeaedd52f58784ea81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:16:47.874750 20917 tablet_service.cc:1467] Processing CreateTablet for tablet 578178d978164b868d6b894b0932325a (DEFAULT_TABLE table=test-table [id=50ab6dfb0f744cbeaedd52f58784ea81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:16:47.874944 20917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578178d978164b868d6b894b0932325a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:47.874944 21044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578178d978164b868d6b894b0932325a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:47.876497 20875 tablet_bootstrap.cc:492] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: Bootstrap starting.
I20240503 08:16:47.876518 21014 tablet_bootstrap.cc:492] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb: Bootstrap starting.
I20240503 08:16:47.876804 20875 tablet_bootstrap.cc:654] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:47.876865 21014 tablet_bootstrap.cc:654] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:47.877377 20875 tablet_bootstrap.cc:492] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: No bootstrap required, opened a new log
I20240503 08:16:47.877440 20875 ts_tablet_manager.cc:1367] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:16:47.877578 20875 raft_consensus.cc:348] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.877655 20875 raft_consensus.cc:374] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:47.877650 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.874552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:36181 (local address 127.0.0.1:59166)
0503 08:16:47.874670 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:47.874682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:47.874724 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:47.875020 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:47.875022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:47.875024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:47.875261 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.875264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.876507 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.876508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:47.877374 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:47.877377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.877464 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.877502 (+ 38us) client_negotiation.cc:769] Sending connection context
0503 08:16:47.877542 (+ 40us) client_negotiation.cc:240] Negotiation successful
0503 08:16:47.877589 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20240503 08:16:47.877676 20875 raft_consensus.cc:729] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23145c47f85941baaf71efc312edbff2, State: Initialized, Role: FOLLOWER
I20240503 08:16:47.877722 20875 consensus_queue.cc:260] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.877815 20875 ts_tablet_manager.cc:1398] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:47.878224 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:47.874650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59166 (local address 127.6.68.195:36181)
0503 08:16:47.874843 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:47.874851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:47.874860 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:16:47.874901 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:47.874902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:47.874919 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:47.875013 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:47.875374 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.876388 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:47.877486 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:47.878003 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:47.878055 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:47.878090 (+ 35us) server_negotiation.cc:299] Negotiation successful
0503 08:16:47.878139 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":122,"threads_started":1}
I20240503 08:16:47.878301 21014 tablet_bootstrap.cc:492] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb: No bootstrap required, opened a new log
I20240503 08:16:47.878352 21014 ts_tablet_manager.cc:1367] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:16:47.878515 21014 raft_consensus.cc:348] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.878602 21014 raft_consensus.cc:374] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:47.878624 21014 raft_consensus.cc:729] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cd69107d6542aa94e6e1f29c43e4bb, State: Initialized, Role: FOLLOWER
I20240503 08:16:47.878683 21014 consensus_queue.cc:260] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.878804 21014 ts_tablet_manager.cc:1398] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:47.878849 21192 tablet_service.cc:1467] Processing CreateTablet for tablet 578178d978164b868d6b894b0932325a (DEFAULT_TABLE table=test-table [id=50ab6dfb0f744cbeaedd52f58784ea81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:16:47.879154 21192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578178d978164b868d6b894b0932325a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:16:47.881769 21267 tablet_bootstrap.cc:492] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: Bootstrap starting.
I20240503 08:16:47.882259 21267 tablet_bootstrap.cc:654] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: Neither blocks nor log segments found. Creating new log.
I20240503 08:16:47.882510 21267 log.cc:824] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:47.883098 21267 tablet_bootstrap.cc:492] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: No bootstrap required, opened a new log
I20240503 08:16:47.883147 21267 ts_tablet_manager.cc:1367] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:47.884862 21267 raft_consensus.cc:348] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.884989 21267 raft_consensus.cc:374] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:16:47.885006 21267 raft_consensus.cc:729] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3385d0203b498d98801e167f26d07b, State: Initialized, Role: FOLLOWER
I20240503 08:16:47.885088 21267 consensus_queue.cc:260] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:47.885331 21267 ts_tablet_manager.cc:1398] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:16:47.885393 21258 heartbeater.cc:503] Master 127.6.68.254:42893 was elected leader, sending a full tablet report...
W20240503 08:16:47.960193 21124 tablet.cc:2369] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:16:48.050742 20989 raft_consensus.cc:482] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:16:48.050894 20989 raft_consensus.cc:504] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:48.051242 20989 leader_election.cc:288] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181), 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643)
I20240503 08:16:48.054067 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.051492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:36181 (local address 127.6.68.193:49343)
0503 08:16:48.051673 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.051683 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:48.051772 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:48.051903 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:48.051904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:48.051905 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:48.052141 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:48.052143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:48.052959 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:48.052960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:48.053771 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:48.053774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:48.053853 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:48.053900 (+ 47us) client_negotiation.cc:769] Sending connection context
0503 08:16:48.053932 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:16:48.053967 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":12,"thread_start_us":101,"threads_started":1}
I20240503 08:16:48.054214 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.051492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38643 (local address 127.6.68.193:52935)
0503 08:16:48.051570 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.051590 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:48.051702 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:48.051903 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:48.051904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:48.051905 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:16:48.052407 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:48.052409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:48.053208 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:48.053209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:48.054006 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:48.054008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:48.054073 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:48.054076 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:16:48.054099 (+ 23us) client_negotiation.cc:240] Negotiation successful
0503 08:16:48.054133 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20240503 08:16:48.054458 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.051595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:49343 (local address 127.6.68.195:36181)
0503 08:16:48.051690 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:48.051693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:48.051769 (+ 76us) server_negotiation.cc:407] Connection header received
0503 08:16:48.051825 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:48.051826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:48.051835 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:48.051895 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:48.052224 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:48.052875 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:48.053887 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:48.054326 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:48.054334 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:48.054357 (+ 23us) server_negotiation.cc:299] Negotiation successful
0503 08:16:48.054388 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20240503 08:16:48.054798 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.051683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:52935 (local address 127.6.68.194:38643)
0503 08:16:48.051726 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:48.051728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:48.051742 (+ 14us) server_negotiation.cc:407] Connection header received
0503 08:16:48.051780 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:48.051781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:48.051813 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:48.051891 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:16:48.052492 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:48.053143 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:48.054141 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:48.054615 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:48.054707 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:48.054728 (+ 21us) server_negotiation.cc:299] Negotiation successful
0503 08:16:48.054754 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240503 08:16:48.054800 21212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578178d978164b868d6b894b0932325a" candidate_uuid: "23145c47f85941baaf71efc312edbff2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3385d0203b498d98801e167f26d07b" is_pre_election: true
I20240503 08:16:48.054939 21212 raft_consensus.cc:2454] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23145c47f85941baaf71efc312edbff2 in term 0.
I20240503 08:16:48.055173 20872 leader_election.cc:302] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23145c47f85941baaf71efc312edbff2, 3a3385d0203b498d98801e167f26d07b; no voters:
I20240503 08:16:48.055235 21077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578178d978164b868d6b894b0932325a" candidate_uuid: "23145c47f85941baaf71efc312edbff2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" is_pre_election: true
I20240503 08:16:48.055289 20989 raft_consensus.cc:2781] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:16:48.055347 20989 raft_consensus.cc:482] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:16:48.055346 21077 raft_consensus.cc:2454] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23145c47f85941baaf71efc312edbff2 in term 0.
I20240503 08:16:48.055366 20989 raft_consensus.cc:3037] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:48.056124 20989 raft_consensus.cc:504] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:48.056253 20989 leader_election.cc:288] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181), 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643)
I20240503 08:16:48.056421 21212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578178d978164b868d6b894b0932325a" candidate_uuid: "23145c47f85941baaf71efc312edbff2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3385d0203b498d98801e167f26d07b"
I20240503 08:16:48.056443 21077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578178d978164b868d6b894b0932325a" candidate_uuid: "23145c47f85941baaf71efc312edbff2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75cd69107d6542aa94e6e1f29c43e4bb"
I20240503 08:16:48.056494 21212 raft_consensus.cc:3037] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:48.056515 21077 raft_consensus.cc:3037] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:16:48.057375 21212 raft_consensus.cc:2454] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23145c47f85941baaf71efc312edbff2 in term 1.
I20240503 08:16:48.057494 21077 raft_consensus.cc:2454] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23145c47f85941baaf71efc312edbff2 in term 1.
I20240503 08:16:48.057576 20872 leader_election.cc:302] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23145c47f85941baaf71efc312edbff2, 3a3385d0203b498d98801e167f26d07b; no voters:
I20240503 08:16:48.057667 20989 raft_consensus.cc:2781] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:16:48.057806 20989 raft_consensus.cc:686] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 LEADER]: Becoming Leader. State: Replica: 23145c47f85941baaf71efc312edbff2, State: Running, Role: LEADER
I20240503 08:16:48.057874 20989 consensus_queue.cc:237] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:48.058611 20795 catalog_manager.cc:5484] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23145c47f85941baaf71efc312edbff2 (127.6.68.193). New cstate: current_term: 1 leader_uuid: "23145c47f85941baaf71efc312edbff2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } health_report { overall_health: HEALTHY } } }
W20240503 08:16:48.066252 20984 tablet.cc:2369] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:16:48.096518 21259 tablet.cc:2369] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:16:48.106928 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.104762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45212 (local address 127.6.68.193:38923)
0503 08:16:48.104884 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:48.104886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:48.104894 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:16:48.104933 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:48.104934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:48.104952 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:48.105024 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:16:48.105264 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:48.105757 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:48.106348 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:48.106479 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:48.106520 (+ 41us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:16:48.106644 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:16:48.106737 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:48.106817 (+ 80us) server_negotiation.cc:299] Negotiation successful
0503 08:16:48.106841 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20240503 08:16:48.108358 21212 raft_consensus.cc:1261] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b [term 1 FOLLOWER]: Refusing update from remote peer 23145c47f85941baaf71efc312edbff2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:16:48.108366 21077 raft_consensus.cc:1261] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Refusing update from remote peer 23145c47f85941baaf71efc312edbff2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:16:48.108680 21271 consensus_queue.cc:1035] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:16:48.108822 20989 consensus_queue.cc:1035] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:16:48.111752 21275 mvcc.cc:205] Tried to move back new op lower bound from 7023510356409401344 to 7023510356205727744. Current Snapshot: MvccSnapshot[applied={T|T < 7023510356409401344 or (T in {7023510356409401344})}]
I20240503 08:16:48.111861 21277 mvcc.cc:205] Tried to move back new op lower bound from 7023510356409401344 to 7023510356205727744. Current Snapshot: MvccSnapshot[applied={T|T < 7023510356409401344}]
I20240503 08:16:48.111968 21276 mvcc.cc:205] Tried to move back new op lower bound from 7023510356409401344 to 7023510356205727744. Current Snapshot: MvccSnapshot[applied={T|T < 7023510356409401344}]
I20240503 08:16:48.113315 20795 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:16:48.113400 20795 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:16:48.114064 20795 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Sending DeleteTablet for 3 replicas of tablet 578178d978164b868d6b894b0932325a
I20240503 08:16:48.114337 21044 tablet_service.cc:1514] Processing DeleteTablet for tablet 578178d978164b868d6b894b0932325a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:16:48 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:34230
I20240503 08:16:48.114403 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 20833
I20240503 08:16:48.114447 21129 tablet_replica.cc:330] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb: stopping tablet replica
I20240503 08:16:48.114477 21192 tablet_service.cc:1514] Processing DeleteTablet for tablet 578178d978164b868d6b894b0932325a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:16:48 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:59166
I20240503 08:16:48.114511 21129 raft_consensus.cc:2229] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:16:48.114552 21129 raft_consensus.cc:2258] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:16:48.114887 21129 ts_tablet_manager.cc:1875] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:16:48.116482 21129 ts_tablet_manager.cc:1888] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:16:48.116542 21129 log.cc:1196] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/578178d978164b868d6b894b0932325a
I20240503 08:16:48.116758 21129 ts_tablet_manager.cc:1909] T 578178d978164b868d6b894b0932325a P 75cd69107d6542aa94e6e1f29c43e4bb: Deleting consensus metadata
I20240503 08:16:48.117455 20779 catalog_manager.cc:4830] TS 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643): tablet 578178d978164b868d6b894b0932325a (table test-table [id=50ab6dfb0f744cbeaedd52f58784ea81]) successfully deleted
W20240503 08:16:48.120093 20779 connection.cc:535] client connection to 127.6.68.193:38923 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:16:48.120186 20779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:16:48.120272 21285 tablet_replica.cc:330] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: stopping tablet replica
I20240503 08:16:48.120291 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 20988
I20240503 08:16:48.120368 21285 raft_consensus.cc:2229] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:16:48.120469 21285 raft_consensus.cc:2258] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:16:48.121081 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.120983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:38923 (local address 127.0.0.1:45226)
0503 08:16:48.121019 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.121053 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20240503 08:16:48.121184 20779 catalog_manager.cc:4573] TS 23145c47f85941baaf71efc312edbff2 (127.6.68.193:38923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 578178d978164b868d6b894b0932325a: Network error: Client connection negotiation failed: client connection to 127.6.68.193:38923: connect: Connection refused (error 111)
I20240503 08:16:48.121779 21285 ts_tablet_manager.cc:1875] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:16:48.123720 21285 ts_tablet_manager.cc:1888] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:16:48.123782 21285 log.cc:1196] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/578178d978164b868d6b894b0932325a
I20240503 08:16:48.124069 21285 ts_tablet_manager.cc:1909] T 578178d978164b868d6b894b0932325a P 3a3385d0203b498d98801e167f26d07b: Deleting consensus metadata
I20240503 08:16:48.124646 20782 catalog_manager.cc:4830] TS 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181): tablet 578178d978164b868d6b894b0932325a (table test-table [id=50ab6dfb0f744cbeaedd52f58784ea81]) successfully deleted
I20240503 08:16:48.126794 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 21127
I20240503 08:16:48.131673 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 20762
I20240503 08:16:48.137300 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:42893
--webserver_interface=127.6.68.254
--webserver_port=44031
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:42893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:48.216361 21287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:48.216565 21287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:48.216588 21287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:48.217931 21287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:16:48.217980 21287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:48.217994 21287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:48.218006 21287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:16:48.218017 21287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:16:48.219668 21287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:42893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:42893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:48.219872 21287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:48.220098 21287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:48.225011 21292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:48.225600 21295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:48.226101 21293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:48.226824 21287 server_base.cc:1034] running on GCE node
I20240503 08:16:48.227048 21287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:48.227278 21287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:48.228422 21287 hybrid_clock.cc:648] HybridClock initialized: now 1714724208228394 us; error 49 us; skew 500 ppm
May 03 08:16:48 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724208, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:48.230907 21287 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:16:48.231374 21287 webserver.cc:460] Webserver started at http://127.6.68.254:44031/ using document root <none> and password file <none>
I20240503 08:16:48.231550 21287 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:48.231595 21287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:48.232884 21287 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:16:48.233543 21302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:48.233704 21287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:16:48.233783 21287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "32a73b315657428d8b198a48d7766b89"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "c2b9766d801f054978b445d5c37c9ffe"
server_key_iv: "5129250375a07c9d6787a9fc6c9ef839"
server_key_version: "encryptionkey@0"
I20240503 08:16:48.234084 21287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:48.249303 21287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:48.251242 21287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:48.251385 21287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:48.255327 21287 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:42893
I20240503 08:16:48.255357 21354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:42893 every 8 connection(s)
I20240503 08:16:48.255736 21287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:16:48.257115 21355 sys_catalog.cc:263] Verifying existing consensus state
I20240503 08:16:48.257668 21355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Bootstrap starting.
I20240503 08:16:48.261773 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 21287
I20240503 08:16:48.262042 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:38923
--local_ip_for_outbound_sockets=127.6.68.193
--tserver_master_addrs=127.6.68.254:42893
--webserver_port=33191
--webserver_interface=127.6.68.193
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:16:48.266963 21355 log.cc:824] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:48.269094 21355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:16:48.269361 21355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89: Bootstrap complete.
I20240503 08:16:48.271440 21355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:48.271653 21355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a73b315657428d8b198a48d7766b89, State: Initialized, Role: FOLLOWER
I20240503 08:16:48.271785 21355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:48.271863 21355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:16:48.271914 21355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:16:48.271965 21355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:16:48.272845 21355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:48.272938 21355 leader_election.cc:302] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32a73b315657428d8b198a48d7766b89; no voters:
I20240503 08:16:48.273109 21355 leader_election.cc:288] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [CANDIDATE]: Term 3 election: Requested vote from peers
I20240503 08:16:48.273157 21359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:16:48.273281 21359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [term 3 LEADER]: Becoming Leader. State: Replica: 32a73b315657428d8b198a48d7766b89, State: Running, Role: LEADER
I20240503 08:16:48.273344 21355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:16:48.273357 21359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } }
I20240503 08:16:48.273603 21359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32a73b315657428d8b198a48d7766b89. Latest consensus state: current_term: 3 leader_uuid: "32a73b315657428d8b198a48d7766b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } } }
I20240503 08:16:48.273705 21359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:48.273612 21360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "32a73b315657428d8b198a48d7766b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a73b315657428d8b198a48d7766b89" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 42893 } } }
I20240503 08:16:48.273871 21360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a73b315657428d8b198a48d7766b89 [sys.catalog]: This master's current role is: LEADER
I20240503 08:16:48.273895 21363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:16:48.274638 21363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:16:48.274745 21363 catalog_manager.cc:1259] Loaded cluster ID: d008473838e547b39221ab937cbf4439
I20240503 08:16:48.274792 21363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20240503 08:16:48.275069 21363 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff469928dcd google::LogMessage::Fail() at ??:0
@ 0x7ff46992cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff4699287cc google::LogMessage::Flush() at ??:0
@ 0x7ff469929f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff46c100af6 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff46c10b294 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff46aa22a08 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff46aa173a4 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff46ad3a6db start_thread at ??:0
@ 0x7ff468e7d71f clone at ??:0
W20240503 08:16:48.360149 21357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:48.360329 21357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:48.360358 21357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:48.361698 21357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:48.361742 21357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:48.361765 21357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:16:48.363291 21357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:38923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=33191
--tserver_master_addrs=127.6.68.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:48.363484 21357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:48.363723 21357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:48.369235 21392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:48.369629 21394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:48.370180 21391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:48.370836 21357 server_base.cc:1034] running on GCE node
I20240503 08:16:48.370996 21357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:48.371196 21357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:48.372332 21357 hybrid_clock.cc:648] HybridClock initialized: now 1714724208372319 us; error 30 us; skew 500 ppm
May 03 08:16:48 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724208, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:48.374851 21357 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:16:48.375298 21357 webserver.cc:460] Webserver started at http://127.6.68.193:33191/ using document root <none> and password file <none>
I20240503 08:16:48.375560 21357 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:48.375653 21357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:48.376915 21357 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:16:48.377532 21401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:48.377763 21357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:16:48.377851 21357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "23145c47f85941baaf71efc312edbff2"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "a8b3ae597c8337a27f13aecfee710e57"
server_key_iv: "5c86af9eae0ba98f095282cdefde528d"
server_key_version: "encryptionkey@0"
I20240503 08:16:48.378132 21357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:48.399523 21357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:48.401798 21357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:48.401927 21357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:48.402149 21357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:48.402554 21408 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 08:16:48.403642 21357 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:16:48.403702 21357 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:16:48.403750 21357 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:16:48.404304 21357 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:16:48.404336 21357 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 08:16:48.404417 21408 tablet_bootstrap.cc:492] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: Bootstrap starting.
I20240503 08:16:48.412101 21357 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:38923
I20240503 08:16:48.412146 21515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:38923 every 8 connection(s)
I20240503 08:16:48.412487 21357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:16:48.416462 21408 log.cc:824] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:16:48.418138 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 21357
I20240503 08:16:48.418418 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:38643
--local_ip_for_outbound_sockets=127.6.68.194
--tserver_master_addrs=127.6.68.254:42893
--webserver_port=41865
--webserver_interface=127.6.68.194
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:16:48.419117 21408 tablet_bootstrap.cc:492] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:16:48.419402 21408 tablet_bootstrap.cc:492] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: Bootstrap complete.
I20240503 08:16:48.420646 21408 ts_tablet_manager.cc:1367] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20240503 08:16:48.422111 21408 raft_consensus.cc:348] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:48.422430 21408 raft_consensus.cc:729] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23145c47f85941baaf71efc312edbff2, State: Initialized, Role: FOLLOWER
I20240503 08:16:48.422574 21408 consensus_queue.cc:260] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:48.422827 21408 ts_tablet_manager.cc:1398] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20240503 08:16:48.465750 21518 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:16:48.413072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:52939)
0503 08:16:48.413231 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.413261 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:48.413721 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:48.465595 (+ 51874us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
W20240503 08:16:48.465996 21516 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:42893 (0 consecutive failures): Network error: Failed to ping master at 127.6.68.254:42893: Client connection negotiation failed: client connection to 127.6.68.254:42893: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:16:48.466408 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.466293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:57257)
0503 08:16:48.466353 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.466387 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20240503 08:16:48.466768 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.466671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:58035)
0503 08:16:48.466723 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.466745 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20240503 08:16:48.466897 21516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240503 08:16:48.500087 21520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:48.500273 21520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:48.500308 21520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:48.501602 21520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:48.501645 21520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:48.501682 21520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:16:48.503170 21520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:38643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=41865
--tserver_master_addrs=127.6.68.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:48.503377 21520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:48.503553 21520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:48.508111 21526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:48.508504 21529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:48.509163 21527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:48.509711 21520 server_base.cc:1034] running on GCE node
I20240503 08:16:48.509896 21520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:48.510115 21520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:48.511265 21520 hybrid_clock.cc:648] HybridClock initialized: now 1714724208511246 us; error 30 us; skew 500 ppm
May 03 08:16:48 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724208, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:48.513818 21520 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:16:48.514251 21520 webserver.cc:460] Webserver started at http://127.6.68.194:41865/ using document root <none> and password file <none>
I20240503 08:16:48.514477 21520 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:48.514525 21520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:48.515748 21520 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:16:48.516255 21536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:48.516438 21520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:48.516528 21520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "75cd69107d6542aa94e6e1f29c43e4bb"
format_stamp: "Formatted at 2024-05-03 08:16:46 on dist-test-slave-3g67"
server_key: "085dac7bd783a7ed1a1a622bb1d8de94"
server_key_iv: "f5d3bee983c67eab029eb814bb7919f4"
server_key_version: "encryptionkey@0"
I20240503 08:16:48.516820 21520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:48.527971 21520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:48.529883 21520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:48.529999 21520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:48.530231 21520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:48.530505 21520 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:48.530537 21520 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:48.530571 21520 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:48.530603 21520 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:48.536528 21520 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:38643
I20240503 08:16:48.536583 21649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:38643 every 8 connection(s)
I20240503 08:16:48.536885 21520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:16:48.537743 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.537458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:48469)
0503 08:16:48.537575 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.537632 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
W20240503 08:16:48.537952 21650 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:42893 (0 consecutive failures): Network error: Failed to ping master at 127.6.68.254:42893: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
I20240503 08:16:48.538177 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.538100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:59303)
0503 08:16:48.538139 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.538161 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20240503 08:16:48.538411 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.538333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:38201)
0503 08:16:48.538369 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.538390 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240503 08:16:48.538499 21650 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240503 08:16:48.543183 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 21520
I20240503 08:16:48.543450 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:36181
--local_ip_for_outbound_sockets=127.6.68.195
--tserver_master_addrs=127.6.68.254:42893
--webserver_port=40801
--webserver_interface=127.6.68.195
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:16:48.621666 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:16:48.621853 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:16:48.621887 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:16:48.623167 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:16:48.623209 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:16:48.623250 21653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:16:48.624779 21653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:36181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=40801
--tserver_master_addrs=127.6.68.254:42893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:16:48.625002 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:16:48.625182 21653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:16:48.629932 21659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:48.630198 21658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:16:48.630856 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:16:48.631381 21653 server_base.cc:1034] running on GCE node
I20240503 08:16:48.631522 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:16:48.631742 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:16:48.632899 21653 hybrid_clock.cc:648] HybridClock initialized: now 1714724208632885 us; error 29 us; skew 500 ppm
May 03 08:16:48 dist-test-slave-3g67 krb5kdc[20217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724208, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:16:48.635244 21653 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:16:48.635699 21653 webserver.cc:460] Webserver started at http://127.6.68.195:40801/ using document root <none> and password file <none>
I20240503 08:16:48.635886 21653 fs_manager.cc:359] Metadata directory not provided
I20240503 08:16:48.635934 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:16:48.637106 21653 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 08:16:48.637681 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:48.637831 21653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:48.637905 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "3a3385d0203b498d98801e167f26d07b"
format_stamp: "Formatted at 2024-05-03 08:16:47 on dist-test-slave-3g67"
server_key: "bcce4ee63d3c3def39ff10de1ae66aa8"
server_key_iv: "982d9806c32c51a9829890b2a1466c6d"
server_key_version: "encryptionkey@0"
I20240503 08:16:48.638190 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:16:48.683125 21653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:16:48.685215 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:16:48.685360 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:16:48.685568 21653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:16:48.685889 21653 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:16:48.685925 21653 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:48.685947 21653 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:16:48.685973 21653 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:16:48.691032 21653 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:36181
I20240503 08:16:48.691090 21781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:36181 every 8 connection(s)
I20240503 08:16:48.691380 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:16:48.692325 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.692031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:44125)
0503 08:16:48.692159 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.692217 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
W20240503 08:16:48.692528 21782 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:42893 (0 consecutive failures): Network error: Failed to ping master at 127.6.68.254:42893: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
I20240503 08:16:48.692799 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.692676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:47781)
0503 08:16:48.692743 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.692776 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20240503 08:16:48.693143 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:48.693029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:43931)
0503 08:16:48.693091 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:48.693122 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20240503 08:16:48.693253 21782 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240503 08:16:48.698293 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 21653
I20240503 08:16:49.467475 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:49.467238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:59399)
0503 08:16:49.467375 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:49.467414 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20240503 08:16:49.539237 21787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:49.538796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:59313)
0503 08:16:49.538940 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:49.538975 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20240503 08:16:49.693878 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:49.693543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:33447)
0503 08:16:49.693667 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:49.693708 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20240503 08:16:50.169338 21789 raft_consensus.cc:482] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:16:50.169504 21789 raft_consensus.cc:504] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:50.169950 21789 leader_election.cc:288] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181), 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643)
May 03 08:16:50 dist-test-slave-3g67 krb5kdc[20217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724208, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.194@KRBTEST.COM
May 03 08:16:50 dist-test-slave-3g67 krb5kdc[20217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724208, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.195@KRBTEST.COM
I20240503 08:16:50.180261 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:50.170135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:38643 (local address 127.6.68.193:40273)
0503 08:16:50.170298 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:50.170306 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:50.170356 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:50.171206 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:50.171209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:50.171835 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:50.172221 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:50.172228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:50.173700 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:50.173701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:50.174278 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:50.174280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:50.174417 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:50.175335 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:50.175339 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:50.176701 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:50.178787 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:50.178793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:50.178796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:50.179120 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:50.179375 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:50.179375 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:50.179376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:50.179424 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:50.179824 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:50.179826 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:50.179898 (+ 72us) client_negotiation.cc:769] Sending connection context
0503 08:16:50.179994 (+ 96us) client_negotiation.cc:240] Negotiation successful
0503 08:16:50.180154 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20240503 08:16:50.180394 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:50.170301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:40273 (local address 127.6.68.194:38643)
0503 08:16:50.170414 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:50.170419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:50.170433 (+ 14us) server_negotiation.cc:407] Connection header received
0503 08:16:50.170929 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:50.170932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:50.170982 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:50.171158 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:50.172361 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:50.173561 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:50.174380 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:50.174636 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:50.176822 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:50.176832 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:50.176832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:50.176854 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:50.178693 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:50.179231 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:50.179235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:50.179235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:50.179302 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:50.179490 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:50.179491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:50.179492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:50.179744 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:50.179842 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:50.180063 (+ 221us) server_negotiation.cc:299] Negotiation successful
0503 08:16:50.180237 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":47,"threads_started":1}
I20240503 08:16:50.180408 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:50.170115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:36181 (local address 127.6.68.193:38291)
0503 08:16:50.170255 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:50.170265 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:16:50.170327 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:16:50.171206 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:16:50.171209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:16:50.171835 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:16:50.172221 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:50.172231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:50.173700 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:16:50.173701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:16:50.174273 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:16:50.174276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:16:50.174416 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:50.175139 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:16:50.175159 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:16:50.177079 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:16:50.178924 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:50.178925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:50.178926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:50.179144 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:50.179344 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:16:50.179345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:16:50.179345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:16:50.179427 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:16:50.179786 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:16:50.179792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:16:50.179900 (+ 108us) client_negotiation.cc:769] Sending connection context
0503 08:16:50.179996 (+ 96us) client_negotiation.cc:240] Negotiation successful
0503 08:16:50.180255 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":115,"thread_start_us":42,"threads_started":1}
I20240503 08:16:50.180799 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:50.170241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:38291 (local address 127.6.68.195:36181)
0503 08:16:50.170374 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0503 08:16:50.170394 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0503 08:16:50.170415 (+ 21us) server_negotiation.cc:407] Connection header received
0503 08:16:50.170930 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:16:50.170933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:16:50.170982 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:16:50.171158 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:16:50.172354 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:50.173573 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:16:50.174379 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:16:50.174633 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:16:50.177173 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:16:50.177181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:16:50.177181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:16:50.177197 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:16:50.178816 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:50.179236 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:50.179238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:50.179238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:50.179282 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:16:50.179495 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:16:50.179496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:16:50.179496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:16:50.179721 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:16:50.179788 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0503 08:16:50.180059 (+ 271us) server_negotiation.cc:299] Negotiation successful
0503 08:16:50.180217 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":49,"threads_started":1}
W20240503 08:16:50.181397 21402 leader_election.cc:341] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643): Not found: Tablet not found: 578178d978164b868d6b894b0932325a
W20240503 08:16:50.181510 21405 leader_election.cc:341] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181): Not found: Tablet not found: 578178d978164b868d6b894b0932325a
I20240503 08:16:50.181583 21405 leader_election.cc:302] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23145c47f85941baaf71efc312edbff2; no voters: 3a3385d0203b498d98801e167f26d07b, 75cd69107d6542aa94e6e1f29c43e4bb
I20240503 08:16:50.181677 21789 raft_consensus.cc:2726] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:16:50.468264 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:50.468115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:50919)
0503 08:16:50.468180 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:50.468222 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20240503 08:16:50.539996 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:50.539785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:34519)
0503 08:16:50.539906 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:50.539953 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20240503 08:16:50.694437 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:50.694269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:47755)
0503 08:16:50.694371 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:50.694400 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20240503 08:16:51.468943 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:51.468718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:48923)
0503 08:16:51.468837 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:51.468876 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20240503 08:16:51.540607 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:51.540423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:39319)
0503 08:16:51.540537 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:51.540566 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20240503 08:16:51.695086 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:51.694891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:36711)
0503 08:16:51.695005 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:51.695043 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20240503 08:16:52.376972 21799 raft_consensus.cc:482] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:16:52.377074 21799 raft_consensus.cc:504] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:52.377265 21799 leader_election.cc:288] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181), 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643)
W20240503 08:16:52.377647 21402 leader_election.cc:341] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643): Not found: Tablet not found: 578178d978164b868d6b894b0932325a
W20240503 08:16:52.377727 21405 leader_election.cc:341] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181): Not found: Tablet not found: 578178d978164b868d6b894b0932325a
I20240503 08:16:52.377761 21405 leader_election.cc:302] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23145c47f85941baaf71efc312edbff2; no voters: 3a3385d0203b498d98801e167f26d07b, 75cd69107d6542aa94e6e1f29c43e4bb
I20240503 08:16:52.377839 21799 raft_consensus.cc:2726] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:16:52.469681 21800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:52.469457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:32909)
0503 08:16:52.469594 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:52.469629 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20240503 08:16:52.541186 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:52.541006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:39737)
0503 08:16:52.541116 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:52.541148 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20240503 08:16:52.695765 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:52.695518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:58737)
0503 08:16:52.695684 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:52.695726 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20240503 08:16:53.470445 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:53.470186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:45139)
0503 08:16:53.470356 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:53.470394 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":63,"threads_started":1}
I20240503 08:16:53.541720 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:53.541545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:54823)
0503 08:16:53.541650 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:53.541678 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20240503 08:16:53.696563 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:53.696282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:35909)
0503 08:16:53.696455 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:53.696501 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
I20240503 08:16:54.140610 21806 raft_consensus.cc:482] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:16:54.140736 21806 raft_consensus.cc:504] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:54.140911 21806 leader_election.cc:288] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181), 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643)
W20240503 08:16:54.141278 21402 leader_election.cc:341] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643): Not found: Tablet not found: 578178d978164b868d6b894b0932325a
W20240503 08:16:54.141371 21405 leader_election.cc:341] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181): Not found: Tablet not found: 578178d978164b868d6b894b0932325a
I20240503 08:16:54.141433 21405 leader_election.cc:302] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23145c47f85941baaf71efc312edbff2; no voters: 3a3385d0203b498d98801e167f26d07b, 75cd69107d6542aa94e6e1f29c43e4bb
I20240503 08:16:54.141498 21806 raft_consensus.cc:2726] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:16:54.471215 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:54.470985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:39229)
0503 08:16:54.471129 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:54.471165 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20240503 08:16:54.542424 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:54.542201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:48951)
0503 08:16:54.542332 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:54.542371 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20240503 08:16:54.697371 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:54.697112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:35853)
0503 08:16:54.697282 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:54.697322 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
I20240503 08:16:55.472098 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:55.471837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:48257)
0503 08:16:55.472004 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:55.472048 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20240503 08:16:55.543139 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:55.542924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:43551)
0503 08:16:55.543052 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:55.543090 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20240503 08:16:55.698045 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:55.697825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:35065)
0503 08:16:55.697963 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:55.697998 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20240503 08:16:56.472952 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:56.472691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:59631)
0503 08:16:56.472828 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:56.472879 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20240503 08:16:56.543921 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:56.543701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:53747)
0503 08:16:56.543829 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:56.543871 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20240503 08:16:56.698741 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:56.698511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:40869)
0503 08:16:56.698645 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:56.698690 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20240503 08:16:57.357642 21816 raft_consensus.cc:482] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:16:57.357743 21816 raft_consensus.cc:504] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:16:57.357889 21816 leader_election.cc:288] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181), 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643)
W20240503 08:16:57.358350 21402 leader_election.cc:341] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643): Not found: Tablet not found: 578178d978164b868d6b894b0932325a
W20240503 08:16:57.358429 21405 leader_election.cc:341] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181): Not found: Tablet not found: 578178d978164b868d6b894b0932325a
I20240503 08:16:57.358471 21405 leader_election.cc:302] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23145c47f85941baaf71efc312edbff2; no voters: 3a3385d0203b498d98801e167f26d07b, 75cd69107d6542aa94e6e1f29c43e4bb
I20240503 08:16:57.358539 21816 raft_consensus.cc:2726] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:16:57.473614 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:57.473392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:50645)
0503 08:16:57.473520 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:57.473555 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20240503 08:16:57.544517 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:57.544315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:43889)
0503 08:16:57.544439 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:57.544471 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20240503 08:16:57.699434 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:57.699205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:47637)
0503 08:16:57.699339 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:57.699388 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20240503 08:16:58.474305 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:58.474065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:53135)
0503 08:16:58.474225 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:58.474256 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20240503 08:16:58.545197 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:58.544975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:51807)
0503 08:16:58.545096 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:58.545124 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20240503 08:16:58.700243 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:58.699995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:56111)
0503 08:16:58.700148 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:58.700192 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20240503 08:16:59.474997 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:59.474773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:41667)
0503 08:16:59.474909 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:59.474945 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20240503 08:16:59.545935 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:59.545703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:39735)
0503 08:16:59.545835 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:59.545883 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20240503 08:16:59.700994 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:16:59.700755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:43431)
0503 08:16:59.700903 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0503 08:16:59.700943 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20240503 08:17:00.475817 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:00.475569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:38753)
0503 08:17:00.475723 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:00.475765 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20240503 08:17:00.546617 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:00.546399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:59151)
0503 08:17:00.546529 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:00.546567 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20240503 08:17:00.701718 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:00.701486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:37169)
0503 08:17:00.701628 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:00.701673 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20240503 08:17:01.476454 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:01.476235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:42637)
0503 08:17:01.476368 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:01.476401 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20240503 08:17:01.547489 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:01.547130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:58231)
0503 08:17:01.547390 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:01.547436 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":93,"threads_started":1}
I20240503 08:17:01.685570 21831 raft_consensus.cc:482] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:17:01.685671 21831 raft_consensus.cc:504] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3385d0203b498d98801e167f26d07b" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 36181 } } peers { permanent_uuid: "75cd69107d6542aa94e6e1f29c43e4bb" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 38643 } } peers { permanent_uuid: "23145c47f85941baaf71efc312edbff2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38923 } }
I20240503 08:17:01.685848 21831 leader_election.cc:288] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181), 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643)
W20240503 08:17:01.686244 21405 leader_election.cc:341] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3a3385d0203b498d98801e167f26d07b (127.6.68.195:36181): Not found: Tablet not found: 578178d978164b868d6b894b0932325a
W20240503 08:17:01.686317 21402 leader_election.cc:341] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75cd69107d6542aa94e6e1f29c43e4bb (127.6.68.194:38643): Not found: Tablet not found: 578178d978164b868d6b894b0932325a
I20240503 08:17:01.686378 21402 leader_election.cc:302] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23145c47f85941baaf71efc312edbff2; no voters: 3a3385d0203b498d98801e167f26d07b, 75cd69107d6542aa94e6e1f29c43e4bb
I20240503 08:17:01.686470 21831 raft_consensus.cc:2726] T 578178d978164b868d6b894b0932325a P 23145c47f85941baaf71efc312edbff2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:17:01.702514 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:01.702283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:37687)
0503 08:17:01.702422 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:01.702462 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
I20240503 08:17:02.477262 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:02.476952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:47505)
0503 08:17:02.477167 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:02.477210 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20240503 08:17:02.548153 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:02.547942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:52297)
0503 08:17:02.548064 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:02.548101 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20240503 08:17:02.703202 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:02.702952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:55437)
0503 08:17:02.703111 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:02.703152 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
I20240503 08:17:03.478101 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:03.477872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.193:45157)
0503 08:17:03.478010 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:03.478053 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20240503 08:17:03.548838 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:03.548640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.194:44355)
0503 08:17:03.548759 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:03.548793 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20240503 08:17:03.700933 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 21357
I20240503 08:17:03.703909 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:03.703733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:42893 (local address 127.6.68.195:38379)
0503 08:17:03.703838 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:03.703867 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
I20240503 08:17:03.705824 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 21520
I20240503 08:17:03.710588 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 21653
2024-05-03T08:17:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18343 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:17:03 dist-test-slave-3g67 krb5kdc[21841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:17:03 dist-test-slave-3g67 krb5kdc[21841](info): set up 2 sockets
May 03 08:17:03 dist-test-slave-3g67 krb5kdc[21841](info): commencing operation
krb5kdc: starting...
W20240503 08:17:04.668037 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:17:04 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:17:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:17:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:17:04.787365 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45119
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:17:04.864312 21854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:04.864503 21854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:04.864539 21854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:04.865772 21854 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240503 08:17:04.865811 21854 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240503 08:17:04.865830 21854 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240503 08:17:04.865849 21854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:17:04.865868 21854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:04.865888 21854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:04.865922 21854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:17:04.865947 21854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:17:04.867555 21854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:45119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:04.867796 21854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:04.867980 21854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:04.872382 21862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:04.873026 21859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:04.873549 21860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:04.874282 21854 server_base.cc:1034] running on GCE node
I20240503 08:17:04.874488 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:04.874687 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:04.875810 21854 hybrid_clock.cc:648] HybridClock initialized: now 1714724224875784 us; error 44 us; skew 500 ppm
May 03 08:17:04 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724224, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:04.878105 21854 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:17:04.878506 21854 webserver.cc:460] Webserver started at http://127.6.68.254:38967/ using document root <none> and password file <none>
I20240503 08:17:04.878679 21854 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:04.878718 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:04.878820 21854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:17:04.879842 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "d1a19a435abd4753b7fe07c8d20396e0"
format_stamp: "Formatted at 2024-05-03 08:17:04 on dist-test-slave-3g67"
server_key: "11e12845bf9546f712183044b43553ac"
server_key_iv: "a28b07935a265f6d67a33326779a0dd0"
server_key_version: "encryptionkey@0"
I20240503 08:17:04.880138 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "d1a19a435abd4753b7fe07c8d20396e0"
format_stamp: "Formatted at 2024-05-03 08:17:04 on dist-test-slave-3g67"
server_key: "11e12845bf9546f712183044b43553ac"
server_key_iv: "a28b07935a265f6d67a33326779a0dd0"
server_key_version: "encryptionkey@0"
I20240503 08:17:04.881300 21854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:17:04.881959 21869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:04.882165 21854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:17:04.882247 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "d1a19a435abd4753b7fe07c8d20396e0"
format_stamp: "Formatted at 2024-05-03 08:17:04 on dist-test-slave-3g67"
server_key: "11e12845bf9546f712183044b43553ac"
server_key_iv: "a28b07935a265f6d67a33326779a0dd0"
server_key_version: "encryptionkey@0"
I20240503 08:17:04.882306 21854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:04.898346 21854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:04.900138 21854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:04.900267 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:04.904000 21854 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:45119
I20240503 08:17:04.904035 21921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:45119 every 8 connection(s)
I20240503 08:17:04.904340 21854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:17:04.905068 21922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:17:04.907099 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Bootstrap starting.
I20240503 08:17:04.907590 21922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Neither blocks nor log segments found. Creating new log.
I20240503 08:17:04.907821 21922 log.cc:824] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:17:04.908385 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: No bootstrap required, opened a new log
I20240503 08:17:04.909739 21922 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:04.909855 21922 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:17:04.909871 21922 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a19a435abd4753b7fe07c8d20396e0, State: Initialized, Role: FOLLOWER
I20240503 08:17:04.909973 21922 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:04.910048 21922 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:17:04.910077 21922 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:17:04.910107 21922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:17:04.910570 21922 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:04.910650 21922 leader_election.cc:302] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a19a435abd4753b7fe07c8d20396e0; no voters:
I20240503 08:17:04.910790 21922 leader_election.cc:288] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:17:04.910851 21925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:17:04.911015 21925 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 1 LEADER]: Becoming Leader. State: Replica: d1a19a435abd4753b7fe07c8d20396e0, State: Running, Role: LEADER
I20240503 08:17:04.911096 21925 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:04.911340 21922 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:17:04.911396 21926 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1a19a435abd4753b7fe07c8d20396e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } } }
I20240503 08:17:04.911464 21926 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:17:04.911634 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 21854
I20240503 08:17:04.911654 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a19a435abd4753b7fe07c8d20396e0. Latest consensus state: current_term: 1 leader_uuid: "d1a19a435abd4753b7fe07c8d20396e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } } }
I20240503 08:17:04.911676 21929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:17:04.911706 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:17:04.911746 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:17:04.911917 6419 external_mini_cluster.cc:1428] Setting key 3bcb026f95bf6cdd38321a6e9e1f7986
I20240503 08:17:04.912098 21929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:17:04.913542 21929 catalog_manager.cc:1347] Generated new cluster ID: 2eb98c8831aa473d9ff8ec2ddc55baf8
I20240503 08:17:04.913590 21929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 03 08:17:04 dist-test-slave-3g67 krb5kdc[21841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:17:04.923120 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:04.912530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.6.68.254:45119)
0503 08:17:04.912693 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:04.912704 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:04.912927 (+ 223us) server_negotiation.cc:407] Connection header received
0503 08:17:04.913023 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:04.913029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:04.913078 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:04.913339 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:17:04.914476 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:04.915674 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:04.916529 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:04.916744 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:04.919201 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:17:04.919207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:17:04.919207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:17:04.919224 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:17:04.921807 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:04.922163 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:04.922165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:04.922166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:04.922222 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:04.922362 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:04.922363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:04.922363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:04.922587 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:17:04.922723 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:04.922799 (+ 76us) server_negotiation.cc:299] Negotiation successful
0503 08:17:04.922918 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20240503 08:17:04.925323 21929 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:17:04.925810 21929 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:17:04.931015 21929 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Generated new TSK 0
I20240503 08:17:04.931171 21929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:17:04.959733 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45119
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:17:05.037288 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:05.037464 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:05.037499 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:05.038756 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:05.038810 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:05.038852 21948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:17:05.040400 21948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:05.040608 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:05.040778 21948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:05.045500 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:05.045783 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:05.046535 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:05.047055 21948 server_base.cc:1034] running on GCE node
I20240503 08:17:05.047231 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:05.047438 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:05.048586 21948 hybrid_clock.cc:648] HybridClock initialized: now 1714724225048552 us; error 49 us; skew 500 ppm
May 03 08:17:05 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724225, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:05.051347 21948 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:17:05.051828 21948 webserver.cc:460] Webserver started at http://127.6.68.193:37447/ using document root <none> and password file <none>
I20240503 08:17:05.052038 21948 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:05.052088 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:05.052194 21948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:17:05.053375 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "4030fdd8a6e340258a8b967b1e6d2287"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "6c7c94cf9411dbd1411ec7630b5c0d8b"
server_key_iv: "ca90f36c7ca8c6fbc49d8107c3dd1438"
server_key_version: "encryptionkey@0"
I20240503 08:17:05.053725 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "4030fdd8a6e340258a8b967b1e6d2287"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "6c7c94cf9411dbd1411ec7630b5c0d8b"
server_key_iv: "ca90f36c7ca8c6fbc49d8107c3dd1438"
server_key_version: "encryptionkey@0"
I20240503 08:17:05.055337 21948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:17:05.056162 21963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:05.056336 21948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:05.056418 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "4030fdd8a6e340258a8b967b1e6d2287"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "6c7c94cf9411dbd1411ec7630b5c0d8b"
server_key_iv: "ca90f36c7ca8c6fbc49d8107c3dd1438"
server_key_version: "encryptionkey@0"
I20240503 08:17:05.056478 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:05.071383 21948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:05.073527 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:05.073626 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:05.073793 21948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:17:05.074038 21948 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:17:05.074065 21948 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:05.074086 21948 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:17:05.074107 21948 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:05.079531 21948 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:33573
I20240503 08:17:05.079591 22076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:33573 every 8 connection(s)
I20240503 08:17:05.079893 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:17:05.084384 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 21948
I20240503 08:17:05.084503 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:17:05.084693 6419 external_mini_cluster.cc:1428] Setting key 4656bee5be3bf1fb6b34ed49217627a1
May 03 08:17:05 dist-test-slave-3g67 krb5kdc[21841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724225, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:17:05.088407 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.080509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:32933 (local address 127.6.68.254:45119)
0503 08:17:05.080562 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:05.080565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:05.080703 (+ 138us) server_negotiation.cc:407] Connection header received
0503 08:17:05.081198 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:05.081199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:05.081213 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:05.081274 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:17:05.082280 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.082768 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.083300 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.083410 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.085630 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:17:05.085632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:17:05.085633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:17:05.085638 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:17:05.087093 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:05.087532 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:05.087532 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:05.087533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:05.087551 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:05.087763 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:05.087763 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:05.087763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:05.087841 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:17:05.087893 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:05.088209 (+ 316us) server_negotiation.cc:299] Negotiation successful
0503 08:17:05.088290 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240503 08:17:05.088531 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.080473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:32933)
0503 08:17:05.080606 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:05.080636 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:05.081107 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:05.081327 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:05.081330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:05.081600 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:17:05.082151 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.082155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.082843 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.082843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:05.083235 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.083236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.083335 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.083892 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:17:05.083907 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:17:05.085541 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:17:05.087171 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:05.087177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:05.087180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:05.087460 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:05.087607 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:05.087608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:05.087608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:05.087708 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:05.087898 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:17:05.087901 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:17:05.088036 (+ 135us) client_negotiation.cc:769] Sending connection context
0503 08:17:05.088159 (+ 123us) client_negotiation.cc:240] Negotiation successful
0503 08:17:05.088309 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20240503 08:17:05.089017 22077 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45119
I20240503 08:17:05.089105 22077 heartbeater.cc:464] Registering TS with master...
I20240503 08:17:05.089298 22077 heartbeater.cc:511] Master 127.6.68.254:45119 requested a full tablet report, sending...
I20240503 08:17:05.089779 21886 ts_manager.cc:194] Registered new tserver with Master: 4030fdd8a6e340258a8b967b1e6d2287 (127.6.68.193:33573)
I20240503 08:17:05.090400 21886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:32933
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:17:05.111361 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45119
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:17:05.189868 22082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:05.190048 22082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:05.190084 22082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:05.191387 22082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:05.191430 22082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:05.191470 22082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:17:05.193001 22082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:05.193239 22082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:05.193413 22082 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:05.197837 22088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:05.198269 22090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:05.198889 22087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:05.199461 22082 server_base.cc:1034] running on GCE node
I20240503 08:17:05.199654 22082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:05.199868 22082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:05.201015 22082 hybrid_clock.cc:648] HybridClock initialized: now 1714724225201004 us; error 29 us; skew 500 ppm
May 03 08:17:05 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724225, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:05.203441 22082 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:17:05.203864 22082 webserver.cc:460] Webserver started at http://127.6.68.194:44489/ using document root <none> and password file <none>
I20240503 08:17:05.204039 22082 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:05.204083 22082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:05.204208 22082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:17:05.205173 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "c10cc5f7280a4652b8bb6aa13401f0a8"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "45716116686e805638208b68731c552b"
server_key_iv: "e4a015e84f8fb7f989982d098d923562"
server_key_version: "encryptionkey@0"
I20240503 08:17:05.205474 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "c10cc5f7280a4652b8bb6aa13401f0a8"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "45716116686e805638208b68731c552b"
server_key_iv: "e4a015e84f8fb7f989982d098d923562"
server_key_version: "encryptionkey@0"
I20240503 08:17:05.206629 22082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240503 08:17:05.207279 22097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:05.207497 22082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:05.207571 22082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "c10cc5f7280a4652b8bb6aa13401f0a8"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "45716116686e805638208b68731c552b"
server_key_iv: "e4a015e84f8fb7f989982d098d923562"
server_key_version: "encryptionkey@0"
I20240503 08:17:05.207654 22082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:05.219481 22082 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:05.221508 22082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:05.221637 22082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:05.221836 22082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:17:05.222115 22082 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:17:05.222149 22082 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:05.222185 22082 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:17:05.222221 22082 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:05.227527 22082 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:41503
I20240503 08:17:05.227581 22210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:41503 every 8 connection(s)
I20240503 08:17:05.227855 22082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
May 03 08:17:05 dist-test-slave-3g67 krb5kdc[21841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724225, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:17:05.235785 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 22082
I20240503 08:17:05.235881 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:17:05.236047 6419 external_mini_cluster.cc:1428] Setting key 6f5b4b3c4244aa7c120aa14259367f01
I20240503 08:17:05.236347 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.228572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:59967 (local address 127.6.68.254:45119)
0503 08:17:05.228652 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:05.228655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:05.228814 (+ 159us) server_negotiation.cc:407] Connection header received
0503 08:17:05.229341 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:05.229342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:05.229357 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:05.229426 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:17:05.230512 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.231037 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.231608 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.231823 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.233767 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:17:05.233774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:17:05.233774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:17:05.233781 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:17:05.235100 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:05.235467 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:05.235469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:05.235469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:05.235494 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:05.235693 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:05.235694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:05.235695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:05.235792 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:17:05.235854 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:05.236120 (+ 266us) server_negotiation.cc:299] Negotiation successful
0503 08:17:05.236209 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240503 08:17:05.236409 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.228565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:59967)
0503 08:17:05.228713 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:05.228737 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:05.229231 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:05.229485 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:05.229490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:05.229789 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:17:05.230409 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.230413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.231140 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.231141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:05.231512 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.231513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.231706 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.232275 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:17:05.232290 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:17:05.233660 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:17:05.235195 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:05.235201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:05.235204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:05.235398 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:05.235551 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:05.235552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:05.235552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:05.235606 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:05.235851 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:17:05.235856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:17:05.235994 (+ 138us) client_negotiation.cc:769] Sending connection context
0503 08:17:05.236093 (+ 99us) client_negotiation.cc:240] Negotiation successful
0503 08:17:05.236221 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20240503 08:17:05.236904 22211 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45119
I20240503 08:17:05.237005 22211 heartbeater.cc:464] Registering TS with master...
I20240503 08:17:05.237248 22211 heartbeater.cc:511] Master 127.6.68.254:45119 requested a full tablet report, sending...
I20240503 08:17:05.237668 21886 ts_manager.cc:194] Registered new tserver with Master: c10cc5f7280a4652b8bb6aa13401f0a8 (127.6.68.194:41503)
I20240503 08:17:05.238070 21886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:59967
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:17:05.263250 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45119
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:17:05.340410 22216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:05.340595 22216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:05.340617 22216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:05.341926 22216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:05.341971 22216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:05.341996 22216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:17:05.343786 22216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:05.343981 22216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:05.344168 22216 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:05.348963 22222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:05.349166 22221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:05.349848 22224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:05.350374 22216 server_base.cc:1034] running on GCE node
I20240503 08:17:05.350528 22216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:05.350733 22216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:05.351871 22216 hybrid_clock.cc:648] HybridClock initialized: now 1714724225351855 us; error 35 us; skew 500 ppm
May 03 08:17:05 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724225, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:05.354307 22216 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:17:05.354734 22216 webserver.cc:460] Webserver started at http://127.6.68.195:39663/ using document root <none> and password file <none>
I20240503 08:17:05.354910 22216 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:05.354954 22216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:05.355058 22216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:17:05.356068 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "03de16e6a5b34772805b33f548747a1d"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "bc548821c5200e825c2c09ee0bf3dc20"
server_key_iv: "b24f070c2ae5d3c24192faa243f8d3ba"
server_key_version: "encryptionkey@0"
I20240503 08:17:05.356367 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "03de16e6a5b34772805b33f548747a1d"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "bc548821c5200e825c2c09ee0bf3dc20"
server_key_iv: "b24f070c2ae5d3c24192faa243f8d3ba"
server_key_version: "encryptionkey@0"
I20240503 08:17:05.357544 22216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:17:05.358215 22231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:05.358400 22216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:17:05.358474 22216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "03de16e6a5b34772805b33f548747a1d"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "bc548821c5200e825c2c09ee0bf3dc20"
server_key_iv: "b24f070c2ae5d3c24192faa243f8d3ba"
server_key_version: "encryptionkey@0"
I20240503 08:17:05.358533 22216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:05.366112 22216 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:05.368141 22216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:05.368264 22216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:05.368440 22216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:17:05.368729 22216 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:17:05.368763 22216 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:05.368798 22216 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:17:05.368829 22216 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:05.374276 22216 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:38601
I20240503 08:17:05.374310 22344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:38601 every 8 connection(s)
I20240503 08:17:05.374577 22216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:17:05.377617 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 22216
I20240503 08:17:05.377703 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:17:05.377868 6419 external_mini_cluster.cc:1428] Setting key 967ea20bef0a24a8760623c421d9f60a
May 03 08:17:05 dist-test-slave-3g67 krb5kdc[21841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724225, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:17:05.382545 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.375252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:47283 (local address 127.6.68.254:45119)
0503 08:17:05.375323 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:05.375325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:05.375438 (+ 113us) server_negotiation.cc:407] Connection header received
0503 08:17:05.375953 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:05.375954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:05.375967 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:05.376017 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:17:05.376976 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.377512 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.378136 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.378286 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.380180 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:17:05.380183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:17:05.380183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:17:05.380189 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:17:05.381565 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:05.381883 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:05.381884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:05.381885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:05.381906 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:05.382065 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:05.382065 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:05.382066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:05.382153 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:17:05.382198 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:05.382420 (+ 222us) server_negotiation.cc:299] Negotiation successful
0503 08:17:05.382483 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240503 08:17:05.382663 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.375195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:47283)
0503 08:17:05.375339 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:05.375366 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:05.375854 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:05.376083 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:05.376086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:05.376381 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:17:05.376879 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.376883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.377594 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.377595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:05.378053 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.378055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.378196 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.378761 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:17:05.378779 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:17:05.380101 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:17:05.381629 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:05.381633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:05.381635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:05.381817 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:05.381963 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:05.381963 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:05.381964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:05.382017 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:05.382196 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:17:05.382199 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:17:05.382299 (+ 100us) client_negotiation.cc:769] Sending connection context
0503 08:17:05.382406 (+ 107us) client_negotiation.cc:240] Negotiation successful
0503 08:17:05.382503 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20240503 08:17:05.383098 22345 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45119
I20240503 08:17:05.383186 22345 heartbeater.cc:464] Registering TS with master...
I20240503 08:17:05.383380 22345 heartbeater.cc:511] Master 127.6.68.254:45119 requested a full tablet report, sending...
I20240503 08:17:05.383786 21886 ts_manager.cc:194] Registered new tserver with Master: 03de16e6a5b34772805b33f548747a1d (127.6.68.195:38601)
I20240503 08:17:05.384140 21886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:47283
I20240503 08:17:05.389627 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:17:05.396414 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.390921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.6.68.254:45119)
0503 08:17:05.390984 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:05.390985 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:05.391031 (+ 46us) server_negotiation.cc:407] Connection header received
0503 08:17:05.391085 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:05.391085 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:05.391093 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:05.391136 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:17:05.391902 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.392406 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.392984 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.393124 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.394082 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:17:05.394085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:17:05.394085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:17:05.394091 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:17:05.395476 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:05.395843 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:05.395844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:05.395844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:05.395865 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:05.396038 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:05.396038 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:05.396038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:05.396134 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:17:05.396208 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:05.396292 (+ 84us) server_negotiation.cc:299] Negotiation successful
0503 08:17:05.396351 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240503 08:17:05.398101 21886 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:17:05.399457 21886 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:17:05.404176 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.401424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:33573 (local address 127.0.0.1:43674)
0503 08:17:05.401678 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:05.401688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:05.401754 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:05.402070 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:05.402073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:05.402074 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:17:05.402290 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.402292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.403262 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.403263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:05.403915 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.403917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.403980 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.404016 (+ 36us) client_negotiation.cc:769] Sending connection context
0503 08:17:05.404065 (+ 49us) client_negotiation.cc:240] Negotiation successful
0503 08:17:05.404109 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":17,"thread_start_us":55,"threads_started":1}
I20240503 08:17:05.404769 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.401518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.6.68.193:33573)
0503 08:17:05.401886 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:05.401895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:05.401907 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:17:05.401949 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:05.401950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:05.401966 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:05.402053 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:17:05.402377 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.403180 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.404011 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.404578 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.404629 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:05.404662 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:17:05.404708 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":84,"threads_started":1}
I20240503 08:17:05.404868 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.401638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:38601 (local address 127.0.0.1:40738)
0503 08:17:05.401770 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:05.401779 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:05.401822 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:05.402221 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:05.402222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:05.402223 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:17:05.402398 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.402400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.403776 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.403776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:05.404638 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.404641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.404715 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.404718 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:17:05.404749 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:17:05.404788 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20240503 08:17:05.404963 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.401424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:41503 (local address 127.0.0.1:47406)
0503 08:17:05.401603 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:05.401626 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:05.401687 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:05.402136 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:05.402137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:05.402140 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:17:05.402326 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.402328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.403757 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.403758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:05.404755 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.404759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.404819 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.404821 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:17:05.404851 (+ 30us) client_negotiation.cc:240] Negotiation successful
0503 08:17:05.404878 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":61,"threads_started":1}
I20240503 08:17:05.405403 22011 tablet_service.cc:1467] Processing CreateTablet for tablet f94b31b122304ac2b8e67cd7120b0a01 (DEFAULT_TABLE table=test-table [id=2af437f2843f4f06ab006620c8121c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:17:05.405510 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.401518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.6.68.194:41503)
0503 08:17:05.401895 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:05.401902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:05.401911 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:17:05.401951 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:05.401952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:05.401969 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:05.402048 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:17:05.402416 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.403563 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.404853 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.405329 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.405369 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:05.405398 (+ 29us) server_negotiation.cc:299] Negotiation successful
0503 08:17:05.405441 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":87,"threads_started":1}
I20240503 08:17:05.405721 22011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94b31b122304ac2b8e67cd7120b0a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:17:05.405907 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.401829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.6.68.195:38601)
0503 08:17:05.402004 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:05.402011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:05.402020 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:17:05.402051 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:05.402053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:05.402067 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:05.402209 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:17:05.402494 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.403604 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.405297 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.405706 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.405745 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:05.405777 (+ 32us) server_negotiation.cc:299] Negotiation successful
0503 08:17:05.405830 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
I20240503 08:17:05.406052 22145 tablet_service.cc:1467] Processing CreateTablet for tablet f94b31b122304ac2b8e67cd7120b0a01 (DEFAULT_TABLE table=test-table [id=2af437f2843f4f06ab006620c8121c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:17:05.406325 22145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94b31b122304ac2b8e67cd7120b0a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:17:05.406405 22279 tablet_service.cc:1467] Processing CreateTablet for tablet f94b31b122304ac2b8e67cd7120b0a01 (DEFAULT_TABLE table=test-table [id=2af437f2843f4f06ab006620c8121c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:17:05.406656 22279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94b31b122304ac2b8e67cd7120b0a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:17:05.408811 22359 tablet_bootstrap.cc:492] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Bootstrap starting.
I20240503 08:17:05.409138 22360 tablet_bootstrap.cc:492] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d: Bootstrap starting.
I20240503 08:17:05.409271 22361 tablet_bootstrap.cc:492] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: Bootstrap starting.
I20240503 08:17:05.409749 22359 tablet_bootstrap.cc:654] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Neither blocks nor log segments found. Creating new log.
I20240503 08:17:05.409837 22360 tablet_bootstrap.cc:654] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d: Neither blocks nor log segments found. Creating new log.
I20240503 08:17:05.409873 22361 tablet_bootstrap.cc:654] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: Neither blocks nor log segments found. Creating new log.
I20240503 08:17:05.410075 22359 log.cc:824] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Log is configured to *not* fsync() on all Append() calls
I20240503 08:17:05.410109 22360 log.cc:824] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:17:05.410152 22361 log.cc:824] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: Log is configured to *not* fsync() on all Append() calls
I20240503 08:17:05.410782 22360 tablet_bootstrap.cc:492] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d: No bootstrap required, opened a new log
I20240503 08:17:05.410782 22359 tablet_bootstrap.cc:492] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: No bootstrap required, opened a new log
I20240503 08:17:05.410851 22359 ts_tablet_manager.cc:1367] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:17:05.410851 22360 ts_tablet_manager.cc:1367] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:17:05.411931 22361 tablet_bootstrap.cc:492] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: No bootstrap required, opened a new log
I20240503 08:17:05.412072 22361 ts_tablet_manager.cc:1367] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:17:05.413185 22360 raft_consensus.cc:348] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } }
I20240503 08:17:05.413185 22359 raft_consensus.cc:348] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } }
I20240503 08:17:05.413321 22360 raft_consensus.cc:374] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:17:05.413321 22359 raft_consensus.cc:374] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:17:05.413347 22359 raft_consensus.cc:729] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4030fdd8a6e340258a8b967b1e6d2287, State: Initialized, Role: FOLLOWER
I20240503 08:17:05.413347 22360 raft_consensus.cc:729] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03de16e6a5b34772805b33f548747a1d, State: Initialized, Role: FOLLOWER
I20240503 08:17:05.413434 22360 consensus_queue.cc:260] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } }
I20240503 08:17:05.413434 22359 consensus_queue.cc:260] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } }
I20240503 08:17:05.413647 22359 ts_tablet_manager.cc:1398] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:17:05.413647 22360 ts_tablet_manager.cc:1398] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:17:05.413784 22345 heartbeater.cc:503] Master 127.6.68.254:45119 was elected leader, sending a full tablet report...
I20240503 08:17:05.414031 22077 heartbeater.cc:503] Master 127.6.68.254:45119 was elected leader, sending a full tablet report...
I20240503 08:17:05.414223 22361 raft_consensus.cc:348] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } }
I20240503 08:17:05.414348 22361 raft_consensus.cc:374] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:17:05.414418 22361 raft_consensus.cc:729] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c10cc5f7280a4652b8bb6aa13401f0a8, State: Initialized, Role: FOLLOWER
I20240503 08:17:05.414535 22361 consensus_queue.cc:260] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } }
I20240503 08:17:05.414777 22361 ts_tablet_manager.cc:1398] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:17:05.414806 22211 heartbeater.cc:503] Master 127.6.68.254:45119 was elected leader, sending a full tablet report...
W20240503 08:17:05.478425 22212 tablet.cc:2369] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:17:05.528074 22366 raft_consensus.cc:482] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:17:05.528232 22366 raft_consensus.cc:504] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } }
I20240503 08:17:05.528589 22366 leader_election.cc:288] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4030fdd8a6e340258a8b967b1e6d2287 (127.6.68.193:33573), c10cc5f7280a4652b8bb6aa13401f0a8 (127.6.68.194:41503)
I20240503 08:17:05.530833 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.528771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:33573 (local address 127.6.68.195:48023)
0503 08:17:05.528832 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:05.528843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:05.528885 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:05.529060 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:05.529061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:05.529063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:17:05.529305 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.529307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.529954 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.529955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:05.530551 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.530552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.530619 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.530623 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:17:05.530655 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:17:05.530690 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20240503 08:17:05.530855 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.528773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:41503 (local address 127.6.68.195:39823)
0503 08:17:05.528956 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:05.528966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:05.529051 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:05.529190 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:05.529191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:05.529192 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:17:05.529381 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.529383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.529976 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.529977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:05.530554 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:05.530556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.530624 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.530628 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:17:05.530659 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:17:05.530692 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":12,"thread_start_us":103,"threads_started":1}
I20240503 08:17:05.531252 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.528841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:48023 (local address 127.6.68.193:33573)
0503 08:17:05.528888 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:05.528890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:05.528925 (+ 35us) server_negotiation.cc:407] Connection header received
0503 08:17:05.528956 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:05.528957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:05.528968 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:05.529044 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:17:05.529391 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.529872 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.530638 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.531152 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.531159 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:05.531177 (+ 18us) server_negotiation.cc:299] Negotiation successful
0503 08:17:05.531207 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:17:05.531271 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.528867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:39823 (local address 127.6.68.194:41503)
0503 08:17:05.528912 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:05.528914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:05.529049 (+ 135us) server_negotiation.cc:407] Connection header received
0503 08:17:05.529112 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:05.529112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:05.529123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:05.529182 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:17:05.529454 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.529904 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.530639 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.531157 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.531165 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:05.531187 (+ 22us) server_negotiation.cc:299] Negotiation successful
0503 08:17:05.531222 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:17:05.531510 22165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94b31b122304ac2b8e67cd7120b0a01" candidate_uuid: "03de16e6a5b34772805b33f548747a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" is_pre_election: true
I20240503 08:17:05.531519 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94b31b122304ac2b8e67cd7120b0a01" candidate_uuid: "03de16e6a5b34772805b33f548747a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4030fdd8a6e340258a8b967b1e6d2287" is_pre_election: true
I20240503 08:17:05.531666 22031 raft_consensus.cc:2454] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03de16e6a5b34772805b33f548747a1d in term 0.
I20240503 08:17:05.531672 22165 raft_consensus.cc:2454] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03de16e6a5b34772805b33f548747a1d in term 0.
I20240503 08:17:05.531921 22234 leader_election.cc:302] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03de16e6a5b34772805b33f548747a1d, c10cc5f7280a4652b8bb6aa13401f0a8; no voters:
I20240503 08:17:05.532083 22366 raft_consensus.cc:2781] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:17:05.532143 22366 raft_consensus.cc:482] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:17:05.532160 22366 raft_consensus.cc:3037] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:17:05.532793 22366 raft_consensus.cc:504] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } }
I20240503 08:17:05.532977 22366 leader_election.cc:288] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [CANDIDATE]: Term 1 election: Requested vote from peers 4030fdd8a6e340258a8b967b1e6d2287 (127.6.68.193:33573), c10cc5f7280a4652b8bb6aa13401f0a8 (127.6.68.194:41503)
I20240503 08:17:05.533069 22165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94b31b122304ac2b8e67cd7120b0a01" candidate_uuid: "03de16e6a5b34772805b33f548747a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8"
I20240503 08:17:05.533069 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94b31b122304ac2b8e67cd7120b0a01" candidate_uuid: "03de16e6a5b34772805b33f548747a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4030fdd8a6e340258a8b967b1e6d2287"
I20240503 08:17:05.533121 22165 raft_consensus.cc:3037] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:17:05.533129 22031 raft_consensus.cc:3037] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:17:05.533731 22165 raft_consensus.cc:2454] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03de16e6a5b34772805b33f548747a1d in term 1.
I20240503 08:17:05.533821 22031 raft_consensus.cc:2454] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03de16e6a5b34772805b33f548747a1d in term 1.
I20240503 08:17:05.533906 22234 leader_election.cc:302] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03de16e6a5b34772805b33f548747a1d, c10cc5f7280a4652b8bb6aa13401f0a8; no voters:
I20240503 08:17:05.534003 22366 raft_consensus.cc:2781] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:17:05.534140 22366 raft_consensus.cc:686] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 1 LEADER]: Becoming Leader. State: Replica: 03de16e6a5b34772805b33f548747a1d, State: Running, Role: LEADER
I20240503 08:17:05.534253 22366 consensus_queue.cc:237] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } }
I20240503 08:17:05.535022 21885 catalog_manager.cc:5484] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 03de16e6a5b34772805b33f548747a1d (127.6.68.195). New cstate: current_term: 1 leader_uuid: "03de16e6a5b34772805b33f548747a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } health_report { overall_health: HEALTHY } } }
I20240503 08:17:05.549165 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.547275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.6.68.195:38601)
0503 08:17:05.547334 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:05.547336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:05.547342 (+ 6us) server_negotiation.cc:407] Connection header received
0503 08:17:05.547372 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:05.547372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:05.547385 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:05.547447 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:17:05.547703 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.548207 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:05.548750 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:05.548877 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:05.548910 (+ 33us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:17:05.548983 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:17:05.549045 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:05.549094 (+ 49us) server_negotiation.cc:299] Negotiation successful
0503 08:17:05.549113 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240503 08:17:05.550706 22031 raft_consensus.cc:1261] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [term 1 FOLLOWER]: Refusing update from remote peer 03de16e6a5b34772805b33f548747a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:17:05.550714 22165 raft_consensus.cc:1261] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 1 FOLLOWER]: Refusing update from remote peer 03de16e6a5b34772805b33f548747a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:17:05.551012 22369 consensus_queue.cc:1035] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:17:05.551111 22366 consensus_queue.cc:1035] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:17:05.555035 22373 mvcc.cc:205] Tried to move back new op lower bound from 7023510427853484032 to 7023510427788890112. Current Snapshot: MvccSnapshot[applied={T|T < 7023510427853484032}]
I20240503 08:17:05.555500 21885 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47242:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:17:05.555595 21885 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47242:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:17:05.556357 21885 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Sending DeleteTablet for 3 replicas of tablet f94b31b122304ac2b8e67cd7120b0a01
I20240503 08:17:05.556674 22145 tablet_service.cc:1514] Processing DeleteTablet for tablet f94b31b122304ac2b8e67cd7120b0a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:17:05 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:47406
I20240503 08:17:05.556694 22279 tablet_service.cc:1514] Processing DeleteTablet for tablet f94b31b122304ac2b8e67cd7120b0a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:17:05 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:40738
I20240503 08:17:05.556735 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 21948
I20240503 08:17:05.556768 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet f94b31b122304ac2b8e67cd7120b0a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:17:05 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:43674
I20240503 08:17:05.558373 22384 tablet_replica.cc:330] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d: stopping tablet replica
I20240503 08:17:05.558471 22384 raft_consensus.cc:2229] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:17:05.558557 22384 raft_consensus.cc:2258] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:17:05.558786 22383 tablet_replica.cc:330] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: stopping tablet replica
I20240503 08:17:05.558921 22383 raft_consensus.cc:2229] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:17:05.559048 22383 raft_consensus.cc:2258] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:17:05.559038 22384 ts_tablet_manager.cc:1875] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:17:05.559500 22383 ts_tablet_manager.cc:1875] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:17:05.560943 22384 ts_tablet_manager.cc:1888] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:17:05.561072 22384 log.cc:1196] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/f94b31b122304ac2b8e67cd7120b0a01
I20240503 08:17:05.561436 22383 ts_tablet_manager.cc:1888] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:17:05.561467 22384 ts_tablet_manager.cc:1909] T f94b31b122304ac2b8e67cd7120b0a01 P 03de16e6a5b34772805b33f548747a1d: Deleting consensus metadata
I20240503 08:17:05.561582 22383 log.cc:1196] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/f94b31b122304ac2b8e67cd7120b0a01
I20240503 08:17:05.561841 21871 catalog_manager.cc:4830] TS 03de16e6a5b34772805b33f548747a1d (127.6.68.195:38601): tablet f94b31b122304ac2b8e67cd7120b0a01 (table test-table [id=2af437f2843f4f06ab006620c8121c66]) successfully deleted
I20240503 08:17:05.561904 22383 ts_tablet_manager.cc:1909] T f94b31b122304ac2b8e67cd7120b0a01 P c10cc5f7280a4652b8bb6aa13401f0a8: Deleting consensus metadata
I20240503 08:17:05.562295 21872 catalog_manager.cc:4830] TS c10cc5f7280a4652b8bb6aa13401f0a8 (127.6.68.194:41503): tablet f94b31b122304ac2b8e67cd7120b0a01 (table test-table [id=2af437f2843f4f06ab006620c8121c66]) successfully deleted
W20240503 08:17:05.562980 21871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:33573 (error 108)
I20240503 08:17:05.563163 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 22082
I20240503 08:17:05.564052 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:05.563939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:33573 (local address 127.0.0.1:43686)
0503 08:17:05.563985 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:05.564016 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20240503 08:17:05.564158 21871 catalog_manager.cc:4573] TS 4030fdd8a6e340258a8b967b1e6d2287 (127.6.68.193:33573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f94b31b122304ac2b8e67cd7120b0a01: Network error: Client connection negotiation failed: client connection to 127.6.68.193:33573: connect: Connection refused (error 111)
I20240503 08:17:05.568902 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 22216
I20240503 08:17:05.573789 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 21854
I20240503 08:17:25.579393 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45119
--webserver_interface=127.6.68.254
--webserver_port=38967
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:17:25.659343 22386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:25.659547 22386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:25.659569 22386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:25.660900 22386 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240503 08:17:25.660969 22386 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240503 08:17:25.660982 22386 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240503 08:17:25.660993 22386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:17:25.661006 22386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:25.661018 22386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:25.661029 22386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:17:25.661041 22386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:17:25.662586 22386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:45119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=38967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:25.662757 22386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:25.662901 22386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:25.670941 22393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:25.671325 22395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:25.672111 22392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:25.672641 22386 server_base.cc:1034] running on GCE node
I20240503 08:17:25.672925 22386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:25.673156 22386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:25.674319 22386 hybrid_clock.cc:648] HybridClock initialized: now 1714724245674326 us; error 54 us; skew 500 ppm
May 03 08:17:25 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724245, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:25.676609 22386 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:17:25.676998 22386 webserver.cc:460] Webserver started at http://127.6.68.254:38967/ using document root <none> and password file <none>
I20240503 08:17:25.677186 22386 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:25.677235 22386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:25.679597 22386 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20240503 08:17:25.680274 22402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:25.680462 22386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:17:25.680567 22386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "d1a19a435abd4753b7fe07c8d20396e0"
format_stamp: "Formatted at 2024-05-03 08:17:04 on dist-test-slave-3g67"
server_key: "11e12845bf9546f712183044b43553ac"
server_key_iv: "a28b07935a265f6d67a33326779a0dd0"
server_key_version: "encryptionkey@0"
I20240503 08:17:25.680907 22386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:25.696353 22386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:25.698362 22386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:25.698493 22386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:25.702663 22386 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:45119
I20240503 08:17:25.702705 22454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:45119 every 8 connection(s)
I20240503 08:17:25.703040 22386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:17:25.704089 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 22386
I20240503 08:17:25.704422 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:33573
--local_ip_for_outbound_sockets=127.6.68.193
--tserver_master_addrs=127.6.68.254:45119
--webserver_port=37447
--webserver_interface=127.6.68.193
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:17:25.704751 22455 sys_catalog.cc:263] Verifying existing consensus state
I20240503 08:17:25.705788 22455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Bootstrap starting.
I20240503 08:17:25.715386 22455 log.cc:824] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:17:25.717389 22455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:17:25.717667 22455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Bootstrap complete.
I20240503 08:17:25.719877 22455 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:25.720067 22455 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a19a435abd4753b7fe07c8d20396e0, State: Initialized, Role: FOLLOWER
I20240503 08:17:25.720182 22455 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:25.720252 22455 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:17:25.720281 22455 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:17:25.720330 22455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:17:25.721217 22455 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:25.721309 22455 leader_election.cc:302] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a19a435abd4753b7fe07c8d20396e0; no voters:
I20240503 08:17:25.721464 22455 leader_election.cc:288] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 08:17:25.721516 22459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:17:25.721676 22459 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 2 LEADER]: Becoming Leader. State: Replica: d1a19a435abd4753b7fe07c8d20396e0, State: Running, Role: LEADER
I20240503 08:17:25.721707 22455 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:17:25.721824 22459 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:25.722245 22461 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a19a435abd4753b7fe07c8d20396e0. Latest consensus state: current_term: 2 leader_uuid: "d1a19a435abd4753b7fe07c8d20396e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } } }
I20240503 08:17:25.722316 22461 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:17:25.722620 22464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:17:25.722659 22460 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d1a19a435abd4753b7fe07c8d20396e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } } }
I20240503 08:17:25.722793 22460 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:17:25.723327 22464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:17:25.723482 22464 catalog_manager.cc:1259] Loaded cluster ID: 2eb98c8831aa473d9ff8ec2ddc55baf8
I20240503 08:17:25.723713 22464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:17:25.724987 22464 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:17:25.725178 22464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Loaded TSK: 0
I20240503 08:17:25.733690 22464 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Generated new TSK 1
I20240503 08:17:25.733799 22464 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240503 08:17:25.787956 22457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:25.788143 22457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:25.788166 22457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:25.789485 22457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:25.789530 22457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:25.789553 22457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:17:25.791177 22457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:33573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=37447
--tserver_master_addrs=127.6.68.254:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:25.791376 22457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:25.791568 22457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:25.796134 22482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:25.796612 22483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:25.797362 22485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:25.797864 22457 server_base.cc:1034] running on GCE node
I20240503 08:17:25.798038 22457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:25.798233 22457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:25.799366 22457 hybrid_clock.cc:648] HybridClock initialized: now 1714724245799355 us; error 31 us; skew 500 ppm
May 03 08:17:25 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724245, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:25.801877 22457 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:17:25.802232 22457 webserver.cc:460] Webserver started at http://127.6.68.193:37447/ using document root <none> and password file <none>
I20240503 08:17:25.802417 22457 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:25.802476 22457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:25.803741 22457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:17:25.804356 22492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:25.804525 22457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:17:25.804601 22457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "4030fdd8a6e340258a8b967b1e6d2287"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "6c7c94cf9411dbd1411ec7630b5c0d8b"
server_key_iv: "ca90f36c7ca8c6fbc49d8107c3dd1438"
server_key_version: "encryptionkey@0"
I20240503 08:17:25.804890 22457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:25.828292 22457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:25.830617 22457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:25.830749 22457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:25.830956 22457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:17:25.831387 22499 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 08:17:25.832531 22457 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:17:25.832576 22457 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:17:25.832616 22457 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:17:25.833454 22457 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:17:25.833487 22457 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 08:17:25.833577 22499 tablet_bootstrap.cc:492] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Bootstrap starting.
I20240503 08:17:25.840607 22457 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:33573
I20240503 08:17:25.840669 22606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:33573 every 8 connection(s)
I20240503 08:17:25.841032 22457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:17:25.846138 22499 log.cc:824] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Log is configured to *not* fsync() on all Append() calls
I20240503 08:17:25.848374 22499 tablet_bootstrap.cc:492] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:17:25.848675 22499 tablet_bootstrap.cc:492] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Bootstrap complete.
May 03 08:17:25 dist-test-slave-3g67 krb5kdc[21841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724245, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:17:25.849601 22499 ts_tablet_manager.cc:1367] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240503 08:17:25.849601 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 22457
I20240503 08:17:25.849956 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:41503
--local_ip_for_outbound_sockets=127.6.68.194
--tserver_master_addrs=127.6.68.254:45119
--webserver_port=44489
--webserver_interface=127.6.68.194
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:17:25.850513 22499 raft_consensus.cc:348] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } }
I20240503 08:17:25.850713 22499 raft_consensus.cc:729] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4030fdd8a6e340258a8b967b1e6d2287, State: Initialized, Role: FOLLOWER
I20240503 08:17:25.850863 22499 consensus_queue.cc:260] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } } peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } }
I20240503 08:17:25.851097 22499 ts_tablet_manager.cc:1398] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 08:17:25.854876 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:25.841651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:33127)
0503 08:17:25.843770 (+ 2119us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:25.843796 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:25.844240 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:25.844757 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:25.844760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:25.845093 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:17:25.845403 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:25.845408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.846450 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:25.846452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:25.847075 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:25.847078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.847213 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:25.848010 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:17:25.848035 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:17:25.851694 (+ 3659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:17:25.853640 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:25.853645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:25.853648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:25.853841 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:25.854024 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:25.854025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:25.854025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:25.854081 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:25.854365 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:17:25.854370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:17:25.854475 (+ 105us) client_negotiation.cc:769] Sending connection context
0503 08:17:25.854567 (+ 92us) client_negotiation.cc:240] Negotiation successful
0503 08:17:25.854713 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2071,"thread_start_us":69,"threads_started":1}
I20240503 08:17:25.854924 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:25.841718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:33127 (local address 127.6.68.254:45119)
0503 08:17:25.844452 (+ 2734us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:25.844460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:25.844479 (+ 19us) server_negotiation.cc:407] Connection header received
0503 08:17:25.844568 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:25.844571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:25.844604 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:25.844720 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:17:25.845523 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.846341 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:25.847178 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.847428 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:25.851821 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:17:25.851846 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:17:25.851846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:17:25.851868 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:17:25.853553 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:25.853917 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:25.853919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:25.853919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:25.853964 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:25.854136 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:25.854137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:25.854137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:25.854302 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:17:25.854381 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:25.854639 (+ 258us) server_negotiation.cc:299] Negotiation successful
0503 08:17:25.854768 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2696,"thread_start_us":77,"threads_started":1}
I20240503 08:17:25.855876 22607 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45119
I20240503 08:17:25.855978 22607 heartbeater.cc:464] Registering TS with master...
I20240503 08:17:25.856145 22607 heartbeater.cc:511] Master 127.6.68.254:45119 requested a full tablet report, sending...
I20240503 08:17:25.856827 22419 ts_manager.cc:194] Registered new tserver with Master: 4030fdd8a6e340258a8b967b1e6d2287 (127.6.68.193:33573)
I20240503 08:17:25.857890 22419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:33127
I20240503 08:17:25.858633 22607 heartbeater.cc:503] Master 127.6.68.254:45119 was elected leader, sending a full tablet report...
I20240503 08:17:25.861856 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:25.857406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:33573 (local address 127.0.0.1:44764)
0503 08:17:25.857590 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:25.857614 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:25.857694 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:25.859916 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:25.859919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:25.859921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:17:25.860139 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:25.860141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.861025 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:25.861026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:25.861579 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:25.861580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.861645 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:25.861672 (+ 27us) client_negotiation.cc:769] Sending connection context
0503 08:17:25.861721 (+ 49us) client_negotiation.cc:240] Negotiation successful
0503 08:17:25.861758 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
I20240503 08:17:25.862320 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:25.857648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.6.68.193:33573)
0503 08:17:25.859757 (+ 2109us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:25.859763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:25.859772 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:17:25.859798 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:25.859800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:25.859813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:25.859901 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:17:25.860214 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.860897 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:25.861657 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.862167 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:25.862212 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:25.862242 (+ 30us) server_negotiation.cc:299] Negotiation successful
0503 08:17:25.862271 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2081,"thread_start_us":88,"threads_started":1}
I20240503 08:17:25.862623 22541 tablet_service.cc:1514] Processing DeleteTablet for tablet f94b31b122304ac2b8e67cd7120b0a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:17:05 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:44764
I20240503 08:17:25.863065 22616 tablet_replica.cc:330] stopping tablet replica
I20240503 08:17:25.863147 22616 raft_consensus.cc:2229] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:17:25.863211 22616 raft_consensus.cc:2258] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:17:25.863538 22616 ts_tablet_manager.cc:1875] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:17:25.865410 22616 ts_tablet_manager.cc:1888] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:17:25.865473 22616 log.cc:1196] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/f94b31b122304ac2b8e67cd7120b0a01
I20240503 08:17:25.865695 22616 ts_tablet_manager.cc:1909] T f94b31b122304ac2b8e67cd7120b0a01 P 4030fdd8a6e340258a8b967b1e6d2287: Deleting consensus metadata
I20240503 08:17:25.866142 22404 catalog_manager.cc:4830] TS 4030fdd8a6e340258a8b967b1e6d2287 (127.6.68.193:33573): tablet f94b31b122304ac2b8e67cd7120b0a01 (table test-table [id=2af437f2843f4f06ab006620c8121c66]) successfully deleted
W20240503 08:17:25.942780 22613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:25.942966 22613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:25.942988 22613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:25.944321 22613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:25.944365 22613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:25.944389 22613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:17:25.945879 22613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:41503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=44489
--tserver_master_addrs=127.6.68.254:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:25.946070 22613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:25.946259 22613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:25.950714 22622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:25.951295 22621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:25.951813 22624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:25.952522 22613 server_base.cc:1034] running on GCE node
I20240503 08:17:25.952702 22613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:25.952914 22613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:25.954061 22613 hybrid_clock.cc:648] HybridClock initialized: now 1714724245954040 us; error 32 us; skew 500 ppm
May 03 08:17:25 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724245, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:25.956614 22613 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:17:25.957031 22613 webserver.cc:460] Webserver started at http://127.6.68.194:44489/ using document root <none> and password file <none>
I20240503 08:17:25.957235 22613 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:25.957295 22613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:25.958554 22613 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:17:25.959077 22631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:25.959255 22613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:17:25.959329 22613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "c10cc5f7280a4652b8bb6aa13401f0a8"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "45716116686e805638208b68731c552b"
server_key_iv: "e4a015e84f8fb7f989982d098d923562"
server_key_version: "encryptionkey@0"
I20240503 08:17:25.959605 22613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:25.977378 22613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:25.979449 22613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:25.979578 22613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:25.979821 22613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:17:25.980109 22613 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:17:25.980144 22613 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:25.980177 22613 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:17:25.980214 22613 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:25.985688 22613 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:41503
I20240503 08:17:25.985749 22744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:41503 every 8 connection(s)
I20240503 08:17:25.986029 22613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
May 03 08:17:25 dist-test-slave-3g67 krb5kdc[21841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724245, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:17:25.993978 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:25.986692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:45123 (local address 127.6.68.254:45119)
0503 08:17:25.986780 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:25.986782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:25.986909 (+ 127us) server_negotiation.cc:407] Connection header received
0503 08:17:25.987352 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:25.987353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:25.987363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:25.987409 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:17:25.988446 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.988933 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:25.989487 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.989605 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:25.991444 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:17:25.991449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:17:25.991450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:17:25.991456 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:17:25.992891 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:25.993246 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:25.993247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:25.993247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:25.993269 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:25.993408 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:25.993409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:25.993409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:25.993492 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:17:25.993538 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:25.993833 (+ 295us) server_negotiation.cc:299] Negotiation successful
0503 08:17:25.993909 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20240503 08:17:25.994102 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:25.986655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:45123)
0503 08:17:25.986805 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:25.986830 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:25.987267 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:25.987458 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:25.987461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:25.987788 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:17:25.988362 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:25.988367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.989002 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:25.989003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:25.989422 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:25.989424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:25.989524 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:25.990003 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:17:25.990016 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:17:25.991355 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:17:25.992971 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:25.992977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:25.992981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:25.993180 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:25.993318 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:25.993319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:25.993319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:25.993366 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:25.993587 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:17:25.993593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:17:25.993715 (+ 122us) client_negotiation.cc:769] Sending connection context
0503 08:17:25.993820 (+ 105us) client_negotiation.cc:240] Negotiation successful
0503 08:17:25.993930 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20240503 08:17:25.994632 22745 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45119
I20240503 08:17:25.994719 22745 heartbeater.cc:464] Registering TS with master...
I20240503 08:17:25.994892 22745 heartbeater.cc:511] Master 127.6.68.254:45119 requested a full tablet report, sending...
I20240503 08:17:25.995306 22419 ts_manager.cc:194] Registered new tserver with Master: c10cc5f7280a4652b8bb6aa13401f0a8 (127.6.68.194:41503)
I20240503 08:17:25.995390 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 22613
I20240503 08:17:25.995662 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:38601
--local_ip_for_outbound_sockets=127.6.68.195
--tserver_master_addrs=127.6.68.254:45119
--webserver_port=39663
--webserver_interface=127.6.68.195
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:17:25.995815 22419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:45123
W20240503 08:17:26.075762 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:26.075953 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:26.075987 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:26.077276 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:26.077320 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:26.077360 22748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:17:26.078924 22748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:38601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=39663
--tserver_master_addrs=127.6.68.254:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:26.079151 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:26.079341 22748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:26.084030 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:26.084408 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:26.084939 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:26.085669 22748 server_base.cc:1034] running on GCE node
I20240503 08:17:26.085863 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:26.086081 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:26.087217 22748 hybrid_clock.cc:648] HybridClock initialized: now 1714724246087195 us; error 38 us; skew 500 ppm
May 03 08:17:26 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724246, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:26.089671 22748 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:17:26.090196 22748 webserver.cc:460] Webserver started at http://127.6.68.195:39663/ using document root <none> and password file <none>
I20240503 08:17:26.090394 22748 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:26.090442 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:26.091738 22748 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:17:26.092239 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:26.092399 22748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:26.092465 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "03de16e6a5b34772805b33f548747a1d"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "bc548821c5200e825c2c09ee0bf3dc20"
server_key_iv: "b24f070c2ae5d3c24192faa243f8d3ba"
server_key_version: "encryptionkey@0"
I20240503 08:17:26.092747 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:26.101938 22748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:26.103502 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:26.103603 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:26.103849 22748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:17:26.104120 22748 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:17:26.104151 22748 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:26.104185 22748 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:17:26.104209 22748 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:26.109731 22748 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:38601
I20240503 08:17:26.109786 22876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:38601 every 8 connection(s)
I20240503 08:17:26.110078 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
May 03 08:17:26 dist-test-slave-3g67 krb5kdc[21841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724246, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:17:26.118155 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.110730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:40829 (local address 127.6.68.254:45119)
0503 08:17:26.110804 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:26.110806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:26.110945 (+ 139us) server_negotiation.cc:407] Connection header received
0503 08:17:26.111431 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:26.111432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:26.111445 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:26.111507 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:17:26.112616 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.113107 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.113640 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.113820 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.115833 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:17:26.115838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:17:26.115838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:17:26.115844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:17:26.117142 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:26.117474 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:26.117475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:26.117476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:26.117497 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:26.117638 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:26.117639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:26.117639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:26.117721 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:17:26.117766 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:26.118006 (+ 240us) server_negotiation.cc:299] Negotiation successful
0503 08:17:26.118088 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240503 08:17:26.118188 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.110689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:40829)
0503 08:17:26.110846 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:26.110869 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:26.111324 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:26.111564 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:26.111567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:26.111991 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:17:26.112512 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:26.112517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.113191 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.113192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:26.113561 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:26.113563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.113692 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.114333 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:17:26.114350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:17:26.115747 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:17:26.117212 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:26.117216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:26.117219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:26.117404 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:26.117552 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:17:26.117553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:17:26.117553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:17:26.117597 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:17:26.117764 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:17:26.117766 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:17:26.117869 (+ 103us) client_negotiation.cc:769] Sending connection context
0503 08:17:26.117952 (+ 83us) client_negotiation.cc:240] Negotiation successful
0503 08:17:26.118052 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20240503 08:17:26.118670 22877 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45119
I20240503 08:17:26.118764 22877 heartbeater.cc:464] Registering TS with master...
I20240503 08:17:26.118954 22877 heartbeater.cc:511] Master 127.6.68.254:45119 requested a full tablet report, sending...
I20240503 08:17:26.119325 22419 ts_manager.cc:194] Registered new tserver with Master: 03de16e6a5b34772805b33f548747a1d (127.6.68.195:38601)
I20240503 08:17:26.119746 22419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:40829
I20240503 08:17:26.120128 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 22748
I20240503 08:17:26.125972 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.120532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.6.68.254:45119)
0503 08:17:26.120585 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:26.120586 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:26.120883 (+ 297us) server_negotiation.cc:407] Connection header received
0503 08:17:26.120904 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:26.120904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:26.120913 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:26.120973 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:17:26.121649 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.122211 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.122769 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.122885 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.123940 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:17:26.123943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:17:26.123943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:17:26.123949 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:17:26.125197 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:26.125500 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:26.125501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:26.125502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:26.125525 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:26.125632 (+ 107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:26.125632 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:26.125633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:26.125711 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:17:26.125815 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:26.125840 (+ 25us) server_negotiation.cc:299] Negotiation successful
0503 08:17:26.125908 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:17:26.126813 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:17:26.129058 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.127159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.6.68.254:45119)
0503 08:17:26.127204 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:26.127206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:26.127307 (+ 101us) server_negotiation.cc:407] Connection header received
0503 08:17:26.127342 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:26.127343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:26.127364 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:26.127428 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:17:26.127719 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.128192 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.128707 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.128819 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.128854 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:17:26.128894 (+ 40us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0503 08:17:26.129004 (+ 110us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50110: token expired
Metrics: {"server-negotiator.queue_time_us":27}
W20240503 08:17:26.129105 22610 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50110: token expired
W20240503 08:17:26.129180 22348 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.68.254:45119)
I20240503 08:17:26.134239 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.129446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50126 (local address 127.6.68.254:45119)
0503 08:17:26.129492 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:26.129494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:26.129500 (+ 6us) server_negotiation.cc:407] Connection header received
0503 08:17:26.129516 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:26.129516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:26.129523 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:26.129577 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:17:26.130117 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.130560 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.131022 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.131127 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.132087 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:17:26.132089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:17:26.132090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:17:26.132095 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:17:26.133330 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:26.133638 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:26.133639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:26.133639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:26.133657 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:17:26.133797 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:17:26.133798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:17:26.133799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:17:26.133907 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:17:26.133991 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:26.134117 (+ 126us) server_negotiation.cc:299] Negotiation successful
0503 08:17:26.134177 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240503 08:17:26.135398 22419 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:17:26.136899 22419 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:17:26.139024 22541 tablet_service.cc:1467] Processing CreateTablet for tablet d8dabab7432640bda7809e4ffca55138 (DEFAULT_TABLE table=test-table [id=02a4e764529c422a9ddfcaca5759be6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:17:26.139227 22541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8dabab7432640bda7809e4ffca55138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:17:26.140708 22499 tablet_bootstrap.cc:492] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: Bootstrap starting.
I20240503 08:17:26.141070 22499 tablet_bootstrap.cc:654] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: Neither blocks nor log segments found. Creating new log.
I20240503 08:17:26.141590 22499 tablet_bootstrap.cc:492] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: No bootstrap required, opened a new log
I20240503 08:17:26.141633 22499 ts_tablet_manager.cc:1367] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:17:26.141652 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.138618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:41503 (local address 127.0.0.1:38248)
0503 08:17:26.138679 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:26.138690 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:26.138732 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:26.139096 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:26.139097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:26.139099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:17:26.139285 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:26.139287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.140565 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.140566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:26.141416 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:26.141419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.141489 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.141493 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:17:26.141529 (+ 36us) client_negotiation.cc:240] Negotiation successful
0503 08:17:26.141574 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20240503 08:17:26.141737 22499 raft_consensus.cc:348] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } } peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } }
I20240503 08:17:26.141810 22499 raft_consensus.cc:374] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:17:26.141839 22499 raft_consensus.cc:729] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4030fdd8a6e340258a8b967b1e6d2287, State: Initialized, Role: FOLLOWER
I20240503 08:17:26.141884 22499 consensus_queue.cc:260] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } } peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } }
I20240503 08:17:26.141963 22499 ts_tablet_manager.cc:1398] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:26.142267 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.138724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.6.68.194:41503)
0503 08:17:26.138923 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:26.138931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:26.138941 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:17:26.138980 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:26.138982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:26.138998 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:26.139100 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:17:26.139368 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.140487 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.141496 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.142037 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.142094 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:26.142128 (+ 34us) server_negotiation.cc:299] Negotiation successful
0503 08:17:26.142184 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":108,"threads_started":1}
I20240503 08:17:26.142515 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.138724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.6.68.195:38601)
0503 08:17:26.138916 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:26.138924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:26.138972 (+ 48us) server_negotiation.cc:407] Connection header received
0503 08:17:26.139057 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:26.139059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:26.139076 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:26.139190 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:17:26.139555 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.140335 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.141701 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.142304 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.142345 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:26.142393 (+ 48us) server_negotiation.cc:299] Negotiation successful
0503 08:17:26.142436 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20240503 08:17:26.141654 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.138618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:38601 (local address 127.0.0.1:47470)
0503 08:17:26.138921 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:26.138931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:26.138985 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:26.139370 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:26.139370 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:26.139371 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:17:26.139485 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:26.139485 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.140426 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.140427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:26.141475 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:26.141477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.141534 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.141537 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:17:26.141568 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:17:26.141594 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":25,"thread_start_us":62,"threads_started":1}
I20240503 08:17:26.142922 22679 tablet_service.cc:1467] Processing CreateTablet for tablet d8dabab7432640bda7809e4ffca55138 (DEFAULT_TABLE table=test-table [id=02a4e764529c422a9ddfcaca5759be6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:17:26.143136 22811 tablet_service.cc:1467] Processing CreateTablet for tablet d8dabab7432640bda7809e4ffca55138 (DEFAULT_TABLE table=test-table [id=02a4e764529c422a9ddfcaca5759be6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:17:26.143230 22679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8dabab7432640bda7809e4ffca55138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:17:26.143373 22811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8dabab7432640bda7809e4ffca55138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:17:26.146524 22886 tablet_bootstrap.cc:492] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d: Bootstrap starting.
I20240503 08:17:26.147050 22886 tablet_bootstrap.cc:654] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d: Neither blocks nor log segments found. Creating new log.
I20240503 08:17:26.147275 22886 log.cc:824] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:17:26.147547 22887 tablet_bootstrap.cc:492] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: Bootstrap starting.
I20240503 08:17:26.148051 22886 tablet_bootstrap.cc:492] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d: No bootstrap required, opened a new log
I20240503 08:17:26.148125 22886 ts_tablet_manager.cc:1367] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:17:26.148249 22887 tablet_bootstrap.cc:654] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: Neither blocks nor log segments found. Creating new log.
I20240503 08:17:26.148577 22887 log.cc:824] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: Log is configured to *not* fsync() on all Append() calls
I20240503 08:17:26.149302 22887 tablet_bootstrap.cc:492] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: No bootstrap required, opened a new log
I20240503 08:17:26.149380 22887 ts_tablet_manager.cc:1367] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:17:26.150481 22886 raft_consensus.cc:348] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } } peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } }
I20240503 08:17:26.150635 22886 raft_consensus.cc:374] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:17:26.150671 22886 raft_consensus.cc:729] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03de16e6a5b34772805b33f548747a1d, State: Initialized, Role: FOLLOWER
I20240503 08:17:26.150772 22886 consensus_queue.cc:260] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } } peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } }
I20240503 08:17:26.151021 22886 ts_tablet_manager.cc:1398] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:17:26.151062 22877 heartbeater.cc:503] Master 127.6.68.254:45119 was elected leader, sending a full tablet report...
I20240503 08:17:26.151589 22887 raft_consensus.cc:348] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } } peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } }
I20240503 08:17:26.151755 22887 raft_consensus.cc:374] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:17:26.151786 22887 raft_consensus.cc:729] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c10cc5f7280a4652b8bb6aa13401f0a8, State: Initialized, Role: FOLLOWER
I20240503 08:17:26.151880 22887 consensus_queue.cc:260] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } } peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } }
I20240503 08:17:26.152108 22887 ts_tablet_manager.cc:1398] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:17:26.152171 22745 heartbeater.cc:503] Master 127.6.68.254:45119 was elected leader, sending a full tablet report...
W20240503 08:17:26.236541 22746 tablet.cc:2369] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:17:26.303602 22890 raft_consensus.cc:482] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:17:26.303751 22890 raft_consensus.cc:504] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } } peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } }
I20240503 08:17:26.304177 22890 leader_election.cc:288] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c10cc5f7280a4652b8bb6aa13401f0a8 (127.6.68.194:41503), 4030fdd8a6e340258a8b967b1e6d2287 (127.6.68.193:33573)
I20240503 08:17:26.307040 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.304360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:33573 (local address 127.6.68.195:53285)
0503 08:17:26.304433 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:26.304447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:26.304503 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:26.304674 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:26.304674 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:26.304676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:17:26.304924 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:26.304926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.305866 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.305867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:26.306804 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:26.306805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.306876 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.306880 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:17:26.306915 (+ 35us) client_negotiation.cc:240] Negotiation successful
0503 08:17:26.306952 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20240503 08:17:26.307049 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.304360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:41503 (local address 127.6.68.195:58651)
0503 08:17:26.304568 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:26.304576 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:26.304623 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:26.304735 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:17:26.304736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:17:26.304737 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:17:26.304970 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:26.304972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.305897 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.305897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:17:26.306795 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:17:26.306798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.306878 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.306880 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:17:26.306915 (+ 35us) client_negotiation.cc:240] Negotiation successful
0503 08:17:26.306952 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":13,"thread_start_us":98,"threads_started":1}
I20240503 08:17:26.307488 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.304485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:53285 (local address 127.6.68.193:33573)
0503 08:17:26.304535 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:26.304537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:26.304549 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:17:26.304582 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:26.304582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:26.304593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:26.304661 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:17:26.305008 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.305779 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.306877 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.307374 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.307382 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:26.307402 (+ 20us) server_negotiation.cc:299] Negotiation successful
0503 08:17:26.307429 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240503 08:17:26.307488 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.304512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:58651 (local address 127.6.68.194:41503)
0503 08:17:26.304557 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:26.304560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:26.304630 (+ 70us) server_negotiation.cc:407] Connection header received
0503 08:17:26.304662 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:26.304663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:26.304672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:26.304725 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:17:26.305034 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.305802 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.306883 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.307354 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.307364 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:26.307388 (+ 24us) server_negotiation.cc:299] Negotiation successful
0503 08:17:26.307422 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:17:26.307783 22547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dabab7432640bda7809e4ffca55138" candidate_uuid: "03de16e6a5b34772805b33f548747a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4030fdd8a6e340258a8b967b1e6d2287" is_pre_election: true
I20240503 08:17:26.307847 22699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dabab7432640bda7809e4ffca55138" candidate_uuid: "03de16e6a5b34772805b33f548747a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" is_pre_election: true
I20240503 08:17:26.307957 22547 raft_consensus.cc:2454] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03de16e6a5b34772805b33f548747a1d in term 0.
I20240503 08:17:26.307967 22699 raft_consensus.cc:2454] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03de16e6a5b34772805b33f548747a1d in term 0.
I20240503 08:17:26.308184 22765 leader_election.cc:302] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03de16e6a5b34772805b33f548747a1d, 4030fdd8a6e340258a8b967b1e6d2287; no voters:
I20240503 08:17:26.308320 22890 raft_consensus.cc:2781] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:17:26.308401 22890 raft_consensus.cc:482] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:17:26.308439 22890 raft_consensus.cc:3037] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:17:26.309129 22890 raft_consensus.cc:504] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } } peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } }
I20240503 08:17:26.309267 22890 leader_election.cc:288] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [CANDIDATE]: Term 1 election: Requested vote from peers c10cc5f7280a4652b8bb6aa13401f0a8 (127.6.68.194:41503), 4030fdd8a6e340258a8b967b1e6d2287 (127.6.68.193:33573)
I20240503 08:17:26.309412 22699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dabab7432640bda7809e4ffca55138" candidate_uuid: "03de16e6a5b34772805b33f548747a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8"
I20240503 08:17:26.309433 22547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dabab7432640bda7809e4ffca55138" candidate_uuid: "03de16e6a5b34772805b33f548747a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4030fdd8a6e340258a8b967b1e6d2287"
I20240503 08:17:26.309479 22699 raft_consensus.cc:3037] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:17:26.309499 22547 raft_consensus.cc:3037] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:17:26.310094 22547 raft_consensus.cc:2454] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03de16e6a5b34772805b33f548747a1d in term 1.
I20240503 08:17:26.310168 22699 raft_consensus.cc:2454] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03de16e6a5b34772805b33f548747a1d in term 1.
I20240503 08:17:26.310263 22765 leader_election.cc:302] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03de16e6a5b34772805b33f548747a1d, 4030fdd8a6e340258a8b967b1e6d2287; no voters:
I20240503 08:17:26.310371 22890 raft_consensus.cc:2781] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:17:26.310513 22890 raft_consensus.cc:686] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 1 LEADER]: Becoming Leader. State: Replica: 03de16e6a5b34772805b33f548747a1d, State: Running, Role: LEADER
I20240503 08:17:26.310603 22890 consensus_queue.cc:237] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } } peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } }
I20240503 08:17:26.311512 22419 catalog_manager.cc:5484] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 03de16e6a5b34772805b33f548747a1d (127.6.68.195). New cstate: current_term: 1 leader_uuid: "03de16e6a5b34772805b33f548747a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03de16e6a5b34772805b33f548747a1d" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 38601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 } health_report { overall_health: UNKNOWN } } }
I20240503 08:17:26.327739 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.323168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.6.68.195:38601)
0503 08:17:26.323220 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:17:26.323222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:17:26.323230 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:17:26.323284 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:17:26.323284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:17:26.323301 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:17:26.323373 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:17:26.325444 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.326296 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:17:26.327079 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:17:26.327269 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:17:26.327324 (+ 55us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:17:26.327442 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:17:26.327508 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0503 08:17:26.327588 (+ 80us) server_negotiation.cc:299] Negotiation successful
0503 08:17:26.327634 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240503 08:17:26.329830 22699 raft_consensus.cc:1261] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 1 FOLLOWER]: Refusing update from remote peer 03de16e6a5b34772805b33f548747a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:17:26.330159 22547 raft_consensus.cc:1261] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287 [term 1 FOLLOWER]: Refusing update from remote peer 03de16e6a5b34772805b33f548747a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:17:26.330305 22890 consensus_queue.cc:1035] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c10cc5f7280a4652b8bb6aa13401f0a8" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:17:26.330487 22893 consensus_queue.cc:1035] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4030fdd8a6e340258a8b967b1e6d2287" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:17:26.332326 22895 mvcc.cc:205] Tried to move back new op lower bound from 7023510512964263936 to 7023510512889352192. Current Snapshot: MvccSnapshot[applied={T|T < 7023510512964263936}]
I20240503 08:17:26.332902 22896 mvcc.cc:205] Tried to move back new op lower bound from 7023510512964263936 to 7023510512889352192. Current Snapshot: MvccSnapshot[applied={T|T < 7023510512964263936}]
I20240503 08:17:26.334690 22419 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50126:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:17:26.334769 22419 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50126:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:17:26.335395 22419 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Sending DeleteTablet for 3 replicas of tablet d8dabab7432640bda7809e4ffca55138
I20240503 08:17:26.335672 22679 tablet_service.cc:1514] Processing DeleteTablet for tablet d8dabab7432640bda7809e4ffca55138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:17:26 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:38248
I20240503 08:17:26.335700 22541 tablet_service.cc:1514] Processing DeleteTablet for tablet d8dabab7432640bda7809e4ffca55138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:17:26 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:44764
I20240503 08:17:26.335783 22616 tablet_replica.cc:330] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: stopping tablet replica
I20240503 08:17:26.335834 22616 raft_consensus.cc:2229] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:17:26.335868 22616 raft_consensus.cc:2258] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:17:26.335964 22811 tablet_service.cc:1514] Processing DeleteTablet for tablet d8dabab7432640bda7809e4ffca55138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:17:26 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:47470
I20240503 08:17:26.336174 22906 tablet_replica.cc:330] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d: stopping tablet replica
I20240503 08:17:26.336252 22905 tablet_replica.cc:330] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: stopping tablet replica
I20240503 08:17:26.336275 22616 ts_tablet_manager.cc:1875] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:17:26.336340 22905 raft_consensus.cc:2229] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:17:26.336289 22906 raft_consensus.cc:2229] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:17:26.336433 22905 raft_consensus.cc:2258] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:17:26.336570 22906 raft_consensus.cc:2258] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:17:26.336867 22905 ts_tablet_manager.cc:1875] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:17:26.337085 22906 ts_tablet_manager.cc:1875] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:17:26.338223 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 22457
I20240503 08:17:26.338580 22905 ts_tablet_manager.cc:1888] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:17:26.338656 22905 log.cc:1196] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/d8dabab7432640bda7809e4ffca55138
I20240503 08:17:26.338914 22906 ts_tablet_manager.cc:1888] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:17:26.338974 22905 ts_tablet_manager.cc:1909] T d8dabab7432640bda7809e4ffca55138 P c10cc5f7280a4652b8bb6aa13401f0a8: Deleting consensus metadata
I20240503 08:17:26.339067 22906 log.cc:1196] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/d8dabab7432640bda7809e4ffca55138
I20240503 08:17:26.339406 22906 ts_tablet_manager.cc:1909] T d8dabab7432640bda7809e4ffca55138 P 03de16e6a5b34772805b33f548747a1d: Deleting consensus metadata
I20240503 08:17:26.339567 22405 catalog_manager.cc:4830] TS c10cc5f7280a4652b8bb6aa13401f0a8 (127.6.68.194:41503): tablet d8dabab7432640bda7809e4ffca55138 (table test-table [id=02a4e764529c422a9ddfcaca5759be6e]) successfully deleted
I20240503 08:17:26.339834 22404 catalog_manager.cc:4830] TS 03de16e6a5b34772805b33f548747a1d (127.6.68.195:38601): tablet d8dabab7432640bda7809e4ffca55138 (table test-table [id=02a4e764529c422a9ddfcaca5759be6e]) successfully deleted
W20240503 08:17:26.344951 22404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:33573 (error 108)
I20240503 08:17:26.345348 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 22613
I20240503 08:17:26.345477 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:26.345331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:33573 (local address 127.0.0.1:44770)
0503 08:17:26.345398 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:26.345437 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20240503 08:17:26.345686 22404 catalog_manager.cc:4573] TS 4030fdd8a6e340258a8b967b1e6d2287 (127.6.68.193:33573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8dabab7432640bda7809e4ffca55138: Network error: Client connection negotiation failed: client connection to 127.6.68.193:33573: connect: Connection refused (error 111)
I20240503 08:17:26.350654 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 22748
I20240503 08:17:26.354846 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 22386
I20240503 08:17:46.360409 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45119
--webserver_interface=127.6.68.254
--webserver_port=38967
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:17:46.438606 22908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:46.438805 22908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:46.438838 22908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:46.440239 22908 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240503 08:17:46.440282 22908 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240503 08:17:46.440304 22908 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240503 08:17:46.440323 22908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:17:46.440353 22908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:46.440380 22908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:46.440405 22908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:17:46.440431 22908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:17:46.442087 22908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:45119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=38967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:46.442308 22908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:46.442492 22908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:46.446931 22914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:46.447584 22916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:46.448103 22913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:46.448801 22908 server_base.cc:1034] running on GCE node
I20240503 08:17:46.449020 22908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:46.449251 22908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:46.450405 22908 hybrid_clock.cc:648] HybridClock initialized: now 1714724266450387 us; error 37 us; skew 500 ppm
May 03 08:17:46 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724266, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:46.452900 22908 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:17:46.453289 22908 webserver.cc:460] Webserver started at http://127.6.68.254:38967/ using document root <none> and password file <none>
I20240503 08:17:46.453440 22908 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:46.453474 22908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:46.454687 22908 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:17:46.455370 22923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:46.455560 22908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:17:46.455677 22908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "d1a19a435abd4753b7fe07c8d20396e0"
format_stamp: "Formatted at 2024-05-03 08:17:04 on dist-test-slave-3g67"
server_key: "11e12845bf9546f712183044b43553ac"
server_key_iv: "a28b07935a265f6d67a33326779a0dd0"
server_key_version: "encryptionkey@0"
I20240503 08:17:46.455955 22908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:46.463959 22908 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:46.466207 22908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:46.466320 22908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:46.470319 22908 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:45119
I20240503 08:17:46.470360 22975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:45119 every 8 connection(s)
I20240503 08:17:46.470671 22908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:17:46.471949 22976 sys_catalog.cc:263] Verifying existing consensus state
I20240503 08:17:46.472482 22976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Bootstrap starting.
I20240503 08:17:46.474850 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 22908
I20240503 08:17:46.475199 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:33573
--local_ip_for_outbound_sockets=127.6.68.193
--tserver_master_addrs=127.6.68.254:45119
--webserver_port=37447
--webserver_interface=127.6.68.193
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:17:46.483973 22976 log.cc:824] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:17:46.486819 22976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:17:46.487146 22976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Bootstrap complete.
I20240503 08:17:46.490456 22976 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:46.490718 22976 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a19a435abd4753b7fe07c8d20396e0, State: Initialized, Role: FOLLOWER
I20240503 08:17:46.490862 22976 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:46.490923 22976 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:17:46.490957 22976 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:17:46.490998 22976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:17:46.492179 22976 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:46.492280 22976 leader_election.cc:302] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a19a435abd4753b7fe07c8d20396e0; no voters:
I20240503 08:17:46.492430 22976 leader_election.cc:288] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20240503 08:17:46.492494 22980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:17:46.492640 22976 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:17:46.492655 22980 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [term 3 LEADER]: Becoming Leader. State: Replica: d1a19a435abd4753b7fe07c8d20396e0, State: Running, Role: LEADER
I20240503 08:17:46.492728 22980 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } }
I20240503 08:17:46.492988 22980 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a19a435abd4753b7fe07c8d20396e0. Latest consensus state: current_term: 3 leader_uuid: "d1a19a435abd4753b7fe07c8d20396e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } } }
I20240503 08:17:46.493088 22980 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:17:46.492988 22981 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d1a19a435abd4753b7fe07c8d20396e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a19a435abd4753b7fe07c8d20396e0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45119 } } }
I20240503 08:17:46.493230 22981 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:17:46.493692 22986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:17:46.494344 22986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:17:46.494491 22986 catalog_manager.cc:1259] Loaded cluster ID: 2eb98c8831aa473d9ff8ec2ddc55baf8
I20240503 08:17:46.494535 22986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:17:46.495344 22986 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:17:46.495509 22986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Loaded TSK: 0
I20240503 08:17:46.495546 22986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1a19a435abd4753b7fe07c8d20396e0: Loaded TSK: 1
F20240503 08:17:46.495759 22986 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7feb748c2dcd google::LogMessage::Fail() at ??:0
@ 0x7feb748c6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7feb748c27cc google::LogMessage::Flush() at ??:0
@ 0x7feb748c3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7feb731142f6 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7feb73111bf3 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7feb770da61e kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7feb770da83a kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7feb770da983 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7feb7709a627 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7feb770a5463 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7feb759bca08 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7feb759b13a4 kudu::Thread::SuperviseThread() at ??:0
@ 0x7feb75cd46db start_thread at ??:0
@ 0x7feb73e1771f clone at ??:0
W20240503 08:17:46.594082 22978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:46.594326 22978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:46.594368 22978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:46.596484 22978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:46.596552 22978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:46.596591 22978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:17:46.598956 22978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:33573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=37447
--tserver_master_addrs=127.6.68.254:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:46.599203 22978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:46.599413 22978 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:46.604581 23017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:46.604995 23020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:46.605574 23018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:46.606186 22978 server_base.cc:1034] running on GCE node
I20240503 08:17:46.606400 22978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:46.606613 22978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:46.607754 22978 hybrid_clock.cc:648] HybridClock initialized: now 1714724266607729 us; error 44 us; skew 500 ppm
May 03 08:17:46 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724266, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:46.610363 22978 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:17:46.610869 22978 webserver.cc:460] Webserver started at http://127.6.68.193:37447/ using document root <none> and password file <none>
I20240503 08:17:46.611101 22978 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:46.611155 22978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:46.612884 22978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:17:46.613566 23027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:46.613729 22978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:17:46.613811 22978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "4030fdd8a6e340258a8b967b1e6d2287"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "6c7c94cf9411dbd1411ec7630b5c0d8b"
server_key_iv: "ca90f36c7ca8c6fbc49d8107c3dd1438"
server_key_version: "encryptionkey@0"
I20240503 08:17:46.614189 22978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:46.666352 22978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:46.668551 22978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:46.668687 22978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:46.668918 22978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:17:46.669380 23034 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
W20240503 08:17:46.670440 23034 ts_tablet_manager.cc:1822] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20240503 08:17:46.670789 23034 ts_tablet_manager.cc:1875] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:17:46.673979 23034 ts_tablet_manager.cc:1888] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:17:46.674070 23034 log.cc:1196] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/wal/wals/d8dabab7432640bda7809e4ffca55138
I20240503 08:17:46.674360 23034 ts_tablet_manager.cc:1909] T d8dabab7432640bda7809e4ffca55138 P 4030fdd8a6e340258a8b967b1e6d2287: Deleting consensus metadata
I20240503 08:17:46.674595 22978 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20240503 08:17:46.674645 22978 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20240503 08:17:46.674690 22978 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:17:46.674718 22978 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:46.680310 22978 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:33573
I20240503 08:17:46.680383 23141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:33573 every 8 connection(s)
I20240503 08:17:46.680657 22978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:17:46.681639 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 22978
I20240503 08:17:46.681941 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:41503
--local_ip_for_outbound_sockets=127.6.68.194
--tserver_master_addrs=127.6.68.254:45119
--webserver_port=44489
--webserver_interface=127.6.68.194
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:17:46.736871 23144 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:17:46.681363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:39717)
0503 08:17:46.681521 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:46.681550 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:17:46.682135 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:17:46.736698 (+ 54563us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
W20240503 08:17:46.737135 23142 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:45119 (0 consecutive failures): Network error: Failed to ping master at 127.6.68.254:45119: Client connection negotiation failed: client connection to 127.6.68.254:45119: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:17:46.737553 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:46.737443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:44117)
0503 08:17:46.737496 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:46.737532 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20240503 08:17:46.737854 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:46.737771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:33443)
0503 08:17:46.737818 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:46.737838 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20240503 08:17:46.737946 23142 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240503 08:17:46.769786 23145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:46.769968 23145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:46.769999 23145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:46.771274 23145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:46.771317 23145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:46.771355 23145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:17:46.772933 23145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:41503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=44489
--tserver_master_addrs=127.6.68.254:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:46.773149 23145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:46.773329 23145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:46.777995 23150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:46.778321 23151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:46.778928 23153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:46.779548 23145 server_base.cc:1034] running on GCE node
I20240503 08:17:46.779747 23145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:46.779965 23145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:46.781113 23145 hybrid_clock.cc:648] HybridClock initialized: now 1714724266781093 us; error 35 us; skew 500 ppm
May 03 08:17:46 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724266, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:46.783434 23145 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:17:46.783835 23145 webserver.cc:460] Webserver started at http://127.6.68.194:44489/ using document root <none> and password file <none>
I20240503 08:17:46.783993 23145 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:46.784029 23145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:46.785167 23145 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:17:46.785709 23160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:46.785907 23145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:17:46.785981 23145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "c10cc5f7280a4652b8bb6aa13401f0a8"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "45716116686e805638208b68731c552b"
server_key_iv: "e4a015e84f8fb7f989982d098d923562"
server_key_version: "encryptionkey@0"
I20240503 08:17:46.786237 23145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:46.814105 23145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:46.815876 23145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:46.815969 23145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:46.816140 23145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:17:46.816421 23145 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:17:46.816454 23145 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:46.816478 23145 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:17:46.816506 23145 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:46.821916 23145 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:41503
I20240503 08:17:46.821982 23273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:41503 every 8 connection(s)
I20240503 08:17:46.822263 23145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:17:46.823155 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:46.822844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:57163)
0503 08:17:46.822981 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:46.823037 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
W20240503 08:17:46.823357 23274 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:45119 (0 consecutive failures): Network error: Failed to ping master at 127.6.68.254:45119: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
I20240503 08:17:46.823599 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:46.823517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:49945)
0503 08:17:46.823558 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:46.823582 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20240503 08:17:46.823864 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:46.823790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:36493)
0503 08:17:46.823827 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:46.823849 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240503 08:17:46.823951 23274 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240503 08:17:46.827337 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 23145
I20240503 08:17:46.827656 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:38601
--local_ip_for_outbound_sockets=127.6.68.195
--tserver_master_addrs=127.6.68.254:45119
--webserver_port=39663
--webserver_interface=127.6.68.195
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:17:46.904786 23277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:17:46.904984 23277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:17:46.905019 23277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:17:46.906316 23277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:17:46.906358 23277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:17:46.906395 23277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:17:46.907948 23277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:38601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=39663
--tserver_master_addrs=127.6.68.254:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:17:46.908178 23277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:17:46.908357 23277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:17:46.912942 23285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:46.913359 23282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:17:46.913865 23283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:17:46.914602 23277 server_base.cc:1034] running on GCE node
I20240503 08:17:46.914774 23277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:17:46.914980 23277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:17:46.916121 23277 hybrid_clock.cc:648] HybridClock initialized: now 1714724266916109 us; error 24 us; skew 500 ppm
May 03 08:17:46 dist-test-slave-3g67 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724266, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:17:46.918637 23277 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:17:46.919071 23277 webserver.cc:460] Webserver started at http://127.6.68.195:39663/ using document root <none> and password file <none>
I20240503 08:17:46.919337 23277 fs_manager.cc:359] Metadata directory not provided
I20240503 08:17:46.919391 23277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:17:46.920712 23277 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:17:46.921253 23292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:46.921408 23277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:17:46.921497 23277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "03de16e6a5b34772805b33f548747a1d"
format_stamp: "Formatted at 2024-05-03 08:17:05 on dist-test-slave-3g67"
server_key: "bc548821c5200e825c2c09ee0bf3dc20"
server_key_iv: "b24f070c2ae5d3c24192faa243f8d3ba"
server_key_version: "encryptionkey@0"
I20240503 08:17:46.921797 23277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:17:46.930115 23277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:17:46.931926 23277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:17:46.932057 23277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:17:46.932242 23277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:17:46.932535 23277 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:17:46.932574 23277 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:46.932610 23277 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:17:46.932649 23277 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:17:46.938300 23277 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:38601
I20240503 08:17:46.938351 23405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:38601 every 8 connection(s)
I20240503 08:17:46.938630 23277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:17:46.939528 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:46.939214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:35583)
0503 08:17:46.939353 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:46.939415 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
W20240503 08:17:46.939757 23406 heartbeater.cc:644] Failed to heartbeat to 127.6.68.254:45119 (0 consecutive failures): Network error: Failed to ping master at 127.6.68.254:45119: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
I20240503 08:17:46.940048 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:46.939930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:34623)
0503 08:17:46.939995 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:46.940026 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20240503 08:17:46.940387 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:46.940298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:34051)
0503 08:17:46.940347 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:46.940370 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20240503 08:17:46.940497 23406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240503 08:17:46.941926 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 23277
I20240503 08:17:47.738574 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:47.738313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:36347)
0503 08:17:47.738459 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:47.738504 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20240503 08:17:47.824522 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:47.824268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:56377)
0503 08:17:47.824393 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:47.824430 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20240503 08:17:47.941107 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:47.940851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:39105)
0503 08:17:47.940988 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:47.941048 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20240503 08:17:48.739303 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:48.739089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:46991)
0503 08:17:48.739215 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:48.739253 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20240503 08:17:48.825129 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:48.824914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:60109)
0503 08:17:48.825020 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:48.825055 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20240503 08:17:48.941736 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:48.941527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:46151)
0503 08:17:48.941663 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:48.941696 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20240503 08:17:49.740017 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:49.739808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:46457)
0503 08:17:49.739930 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:49.739966 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20240503 08:17:49.825819 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:49.825606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:58389)
0503 08:17:49.825736 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:49.825774 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20240503 08:17:49.942507 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:49.942237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:57443)
0503 08:17:49.942423 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:49.942460 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
I20240503 08:17:50.740666 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:50.740465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:43011)
0503 08:17:50.740578 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:50.740604 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20240503 08:17:50.826555 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:50.826323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:42231)
0503 08:17:50.826472 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:50.826509 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20240503 08:17:50.943256 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:50.942984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:51815)
0503 08:17:50.943167 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:50.943202 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
I20240503 08:17:51.741405 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:51.741175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:44893)
0503 08:17:51.741311 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:51.741355 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20240503 08:17:51.827265 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:51.827039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:48875)
0503 08:17:51.827173 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:51.827209 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20240503 08:17:51.943938 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:51.943736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:33117)
0503 08:17:51.943861 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:51.943896 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20240503 08:17:52.742192 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:52.741971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:47335)
0503 08:17:52.742096 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:52.742137 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20240503 08:17:52.828030 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:52.827792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:38335)
0503 08:17:52.827937 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:52.827981 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20240503 08:17:52.944837 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:52.944514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:56325)
0503 08:17:52.944711 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:52.944771 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":107,"threads_started":1}
I20240503 08:17:53.743021 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:53.742770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:39303)
0503 08:17:53.742920 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:53.742970 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20240503 08:17:53.828809 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:53.828586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:55451)
0503 08:17:53.828713 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:53.828757 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20240503 08:17:53.945636 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:53.945367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:59811)
0503 08:17:53.945540 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:53.945580 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20240503 08:17:54.743906 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:54.743694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:48577)
0503 08:17:54.743817 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:54.743856 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20240503 08:17:54.829474 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:54.829260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:48735)
0503 08:17:54.829396 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:54.829430 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20240503 08:17:54.946372 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:54.946152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:50277)
0503 08:17:54.946279 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:54.946310 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20240503 08:17:55.744604 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:55.744365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:45435)
0503 08:17:55.744507 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:55.744546 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20240503 08:17:55.830166 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:55.829925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:53485)
0503 08:17:55.830071 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:55.830114 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":59,"threads_started":1}
I20240503 08:17:55.947052 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:55.946829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:33675)
0503 08:17:55.946963 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:55.946997 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20240503 08:17:56.745309 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:56.745087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:48373)
0503 08:17:56.745220 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:56.745258 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20240503 08:17:56.830942 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:56.830729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:53983)
0503 08:17:56.830861 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:56.830896 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20240503 08:17:56.947763 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:56.947487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:54245)
0503 08:17:56.947616 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:56.947715 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20240503 08:17:57.746013 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:57.745788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:59889)
0503 08:17:57.745928 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:57.745962 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20240503 08:17:57.831614 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:57.831391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:38897)
0503 08:17:57.831531 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:57.831565 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20240503 08:17:57.948525 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:57.948263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:49047)
0503 08:17:57.948431 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:57.948473 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20240503 08:17:58.746753 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:58.746541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:47497)
0503 08:17:58.746666 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:58.746703 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20240503 08:17:58.832302 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:58.832083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:38047)
0503 08:17:58.832203 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:58.832237 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20240503 08:17:58.949299 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:58.949072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:51515)
0503 08:17:58.949201 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:58.949248 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20240503 08:17:59.747534 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:59.747274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:46267)
0503 08:17:59.747424 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:59.747480 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20240503 08:17:59.833202 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:59.832980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:37579)
0503 08:17:59.833104 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:59.833144 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20240503 08:17:59.950047 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:17:59.949821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:58065)
0503 08:17:59.949951 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0503 08:17:59.949997 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20240503 08:18:00.748365 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:00.748049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:52831)
0503 08:18:00.748232 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:00.748277 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20240503 08:18:00.833878 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:00.833655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:58143)
0503 08:18:00.833795 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:00.833832 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20240503 08:18:00.950803 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:00.950555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:37819)
0503 08:18:00.950715 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:00.950753 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20240503 08:18:01.749215 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:01.748976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.193:38695)
0503 08:18:01.749125 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:01.749165 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20240503 08:18:01.834656 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:01.834386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.194:46237)
0503 08:18:01.834558 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:01.834604 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20240503 08:18:01.946118 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 22978
I20240503 08:18:01.951392 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 23145
I20240503 08:18:01.951655 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:01.951400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45119 (local address 127.6.68.195:37661)
0503 08:18:01.951541 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:01.951575 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.254:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20240503 08:18:01.955940 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 23277
2024-05-03T08:18:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58245 ms)
[----------] 30 tests from SecurityITest (268163 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:02 dist-test-slave-3g67 krb5kdc[23457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:02 dist-test-slave-3g67 krb5kdc[23457](info): set up 2 sockets
May 03 08:18:02 dist-test-slave-3g67 krb5kdc[23457](info): commencing operation
krb5kdc: starting...
W20240503 08:18:02.923765 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:03 dist-test-slave-3g67 krb5kdc[23457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:18:03.040516 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45221
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:45221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:03.119084 23470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:03.119278 23470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:03.119311 23470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:03.120643 23470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:03.120692 23470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:03.120714 23470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:03.120735 23470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:03.120754 23470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:03.122364 23470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:45221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:03.122624 23470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:03.122802 23470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:03.127444 23476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:03.128000 23478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:03.128518 23475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:03.129168 23470 server_base.cc:1034] running on GCE node
I20240503 08:18:03.129393 23470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:03.129626 23470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:03.130770 23470 hybrid_clock.cc:648] HybridClock initialized: now 1714724283130695 us; error 91 us; skew 500 ppm
May 03 08:18:03 dist-test-slave-3g67 krb5kdc[23457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:03.133153 23470 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:18:03.133538 23470 webserver.cc:460] Webserver started at http://127.6.68.254:46009/ using document root <none> and password file <none>
I20240503 08:18:03.133724 23470 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:03.133766 23470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:03.133867 23470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:03.134905 23470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "e0f951be647b4fa683163cb8c54de326"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "d7a0508f5e5bffbeb4d2f9e48ab2ce04"
server_key_iv: "f3e4cfa6a83c057911a334607356b429"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.135210 23470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "e0f951be647b4fa683163cb8c54de326"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "d7a0508f5e5bffbeb4d2f9e48ab2ce04"
server_key_iv: "f3e4cfa6a83c057911a334607356b429"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.136476 23470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:18:03.137183 23485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:03.137355 23470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:03.137431 23470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "e0f951be647b4fa683163cb8c54de326"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "d7a0508f5e5bffbeb4d2f9e48ab2ce04"
server_key_iv: "f3e4cfa6a83c057911a334607356b429"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.137499 23470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:03.153980 23470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:03.155788 23470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:03.155906 23470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:03.159469 23470 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:45221
I20240503 08:18:03.159518 23537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:45221 every 8 connection(s)
I20240503 08:18:03.159853 23470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:03.160570 23538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:03.162611 23538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326: Bootstrap starting.
I20240503 08:18:03.163146 23538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:03.163399 23538 log.cc:824] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:03.164068 23538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326: No bootstrap required, opened a new log
I20240503 08:18:03.164860 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 23470
I20240503 08:18:03.164948 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:03.165129 6419 external_mini_cluster.cc:1428] Setting key fd8a7aa57471d5949ef8d3cea098e42e
I20240503 08:18:03.165781 23538 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f951be647b4fa683163cb8c54de326" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45221 } }
I20240503 08:18:03.165936 23538 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:03.165962 23538 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f951be647b4fa683163cb8c54de326, State: Initialized, Role: FOLLOWER
I20240503 08:18:03.166062 23538 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f951be647b4fa683163cb8c54de326" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45221 } }
I20240503 08:18:03.166138 23538 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:03.166205 23538 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:03.166254 23538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:03.166779 23538 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f951be647b4fa683163cb8c54de326" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45221 } }
I20240503 08:18:03.166863 23538 leader_election.cc:302] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0f951be647b4fa683163cb8c54de326; no voters:
I20240503 08:18:03.166997 23538 leader_election.cc:288] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:03.167199 23543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:03.167253 23538 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:03.167320 23543 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [term 1 LEADER]: Becoming Leader. State: Replica: e0f951be647b4fa683163cb8c54de326, State: Running, Role: LEADER
I20240503 08:18:03.167398 23543 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f951be647b4fa683163cb8c54de326" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45221 } }
I20240503 08:18:03.167721 23543 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0f951be647b4fa683163cb8c54de326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f951be647b4fa683163cb8c54de326" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45221 } } }
I20240503 08:18:03.167893 23543 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:03.168097 23543 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f951be647b4fa683163cb8c54de326. Latest consensus state: current_term: 1 leader_uuid: "e0f951be647b4fa683163cb8c54de326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f951be647b4fa683163cb8c54de326" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45221 } } }
I20240503 08:18:03.168162 23543 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:03.168392 23552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:03.168905 23552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:03.170557 23552 catalog_manager.cc:1347] Generated new cluster ID: c22866d66452463e84283bd74dce62d8
I20240503 08:18:03.171160 23552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 03 08:18:03 dist-test-slave-3g67 krb5kdc[23457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:03.175364 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.165647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.6.68.254:45221)
0503 08:18:03.165835 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:03.165846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:03.165869 (+ 23us) server_negotiation.cc:407] Connection header received
0503 08:18:03.165966 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:03.165972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:03.166029 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:03.166173 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:03.167008 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.168383 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.169164 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.169425 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.171954 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:03.171978 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:03.171978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:03.171992 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:03.173572 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:03.173906 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:03.173909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:03.173910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:03.173971 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:03.174147 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:03.174148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:03.174148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:03.174390 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:03.174538 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:03.175094 (+ 556us) server_negotiation.cc:299] Negotiation successful
0503 08:18:03.175219 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20240503 08:18:03.195192 23552 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:03.195796 23552 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:03.202800 23552 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326: Generated new TSK 0
I20240503 08:18:03.202986 23552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:18:03.234459 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45221
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:03.313352 23564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:03.313544 23564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:03.313581 23564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:03.314860 23564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:03.314903 23564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:03.314939 23564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:18:03.316555 23564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:03.316776 23564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:03.316953 23564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:03.321631 23570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:03.322218 23572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:03.322751 23569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:03.323413 23564 server_base.cc:1034] running on GCE node
I20240503 08:18:03.323587 23564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:03.323849 23564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:03.324988 23564 hybrid_clock.cc:648] HybridClock initialized: now 1714724283324978 us; error 28 us; skew 500 ppm
May 03 08:18:03 dist-test-slave-3g67 krb5kdc[23457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:03.327912 23564 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:18:03.328397 23564 webserver.cc:460] Webserver started at http://127.6.68.193:46819/ using document root <none> and password file <none>
I20240503 08:18:03.328598 23564 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:03.328644 23564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:03.328747 23564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:03.329954 23564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "b1ed3eec128a47b6978d93a8d2568956"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "8d74592628b94ce2697a8f95f9acf82c"
server_key_iv: "93a6ef72a1f344377e59580de27b9eeb"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.330333 23564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "b1ed3eec128a47b6978d93a8d2568956"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "8d74592628b94ce2697a8f95f9acf82c"
server_key_iv: "93a6ef72a1f344377e59580de27b9eeb"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.331537 23564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:18:03.332268 23579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:03.332460 23564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:03.332535 23564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "b1ed3eec128a47b6978d93a8d2568956"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "8d74592628b94ce2697a8f95f9acf82c"
server_key_iv: "93a6ef72a1f344377e59580de27b9eeb"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.332595 23564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:03.361217 23564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:03.363106 23564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:03.363224 23564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:03.363444 23564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:03.363761 23564 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:03.363798 23564 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:03.363832 23564 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:03.363862 23564 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:03.369632 23564 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:33591
I20240503 08:18:03.369686 23692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:33591 every 8 connection(s)
I20240503 08:18:03.369951 23564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
May 03 08:18:03 dist-test-slave-3g67 krb5kdc[23457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:03.377858 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.370576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:53623 (local address 127.6.68.254:45221)
0503 08:18:03.370633 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:03.370635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:03.370800 (+ 165us) server_negotiation.cc:407] Connection header received
0503 08:18:03.371233 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:03.371233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:03.371245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:03.371306 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:03.372057 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.372554 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.373500 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.373618 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.375544 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:03.375549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:03.375549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:03.375555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:03.376889 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:03.377216 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:03.377217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:03.377218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:03.377233 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:03.377389 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:03.377390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:03.377390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:03.377469 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:03.377517 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:03.377714 (+ 197us) server_negotiation.cc:299] Negotiation successful
0503 08:18:03.377785 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240503 08:18:03.377946 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.370531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45221 (local address 127.6.68.193:53623)
0503 08:18:03.370689 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:03.370719 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:03.371154 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:03.371357 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:03.371361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:03.371690 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:03.371970 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.371973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.372636 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.372637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:03.373428 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.373431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.373537 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.374064 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:03.374080 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:03.375433 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:03.376962 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:03.376967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:03.376969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:03.377154 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:03.377298 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:03.377299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:03.377299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:03.377346 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:03.377515 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:03.377519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:03.377616 (+ 97us) client_negotiation.cc:769] Sending connection context
0503 08:18:03.377697 (+ 81us) client_negotiation.cc:240] Negotiation successful
0503 08:18:03.377796 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20240503 08:18:03.378432 23693 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45221
I20240503 08:18:03.378528 23693 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:03.378711 23693 heartbeater.cc:511] Master 127.6.68.254:45221 requested a full tablet report, sending...
I20240503 08:18:03.379141 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 23564
I20240503 08:18:03.379240 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:18:03.379212 23502 ts_manager.cc:194] Registered new tserver with Master: b1ed3eec128a47b6978d93a8d2568956 (127.6.68.193:33591)
I20240503 08:18:03.379418 6419 external_mini_cluster.cc:1428] Setting key a75e730c029366c84350a5bfd386d206
I20240503 08:18:03.379976 23502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:53623
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:18:03.402913 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45221
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:03.483309 23698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:03.483480 23698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:03.483503 23698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:03.484819 23698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:03.484864 23698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:03.484889 23698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:18:03.486459 23698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:03.486653 23698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:03.486840 23698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:03.491384 23704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:03.491839 23706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:03.492314 23703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:03.492985 23698 server_base.cc:1034] running on GCE node
I20240503 08:18:03.493161 23698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:03.493382 23698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:03.494555 23698 hybrid_clock.cc:648] HybridClock initialized: now 1714724283494532 us; error 34 us; skew 500 ppm
May 03 08:18:03 dist-test-slave-3g67 krb5kdc[23457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:03.496922 23698 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:18:03.497314 23698 webserver.cc:460] Webserver started at http://127.6.68.194:37623/ using document root <none> and password file <none>
I20240503 08:18:03.497470 23698 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:03.497505 23698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:03.497586 23698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:03.498533 23698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "8c4f22276abb4a83b005e3c2cd020be5"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "e9d933948e91b8e5d45c760e0c135734"
server_key_iv: "82b0b26dd2538c51b32e0ee490f1c489"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.498814 23698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "8c4f22276abb4a83b005e3c2cd020be5"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "e9d933948e91b8e5d45c760e0c135734"
server_key_iv: "82b0b26dd2538c51b32e0ee490f1c489"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.499944 23698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:03.500599 23713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:03.500764 23698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:03.500829 23698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "8c4f22276abb4a83b005e3c2cd020be5"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "e9d933948e91b8e5d45c760e0c135734"
server_key_iv: "82b0b26dd2538c51b32e0ee490f1c489"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.500900 23698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:03.519681 23698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:03.521332 23698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:03.521432 23698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:03.521629 23698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:03.521912 23698 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:03.521944 23698 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:03.521967 23698 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:03.522001 23698 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:03.527108 23698 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:40619
I20240503 08:18:03.527149 23826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:40619 every 8 connection(s)
I20240503 08:18:03.527421 23698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:18:03.527612 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 23698
I20240503 08:18:03.527743 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:18:03.527909 6419 external_mini_cluster.cc:1428] Setting key c3f319bea4bb92cffe765c2426397d1e
May 03 08:18:03 dist-test-slave-3g67 krb5kdc[23457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
I20240503 08:18:03.536367 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.528145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:33463 (local address 127.6.68.254:45221)
0503 08:18:03.528193 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:03.528196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:03.528415 (+ 219us) server_negotiation.cc:407] Connection header received
0503 08:18:03.528926 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:03.528928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:03.528940 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:03.528993 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:03.529695 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.530290 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.531253 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.531470 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.533310 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:03.533314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:03.533315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:03.533323 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:03.535101 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:03.535503 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:03.535505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:03.535505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:03.535552 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:03.535763 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:03.535764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:03.535764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:03.535858 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:03.535919 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:03.536160 (+ 241us) server_negotiation.cc:299] Negotiation successful
0503 08:18:03.536262 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240503 08:18:03.536448 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.528144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45221 (local address 127.6.68.194:33463)
0503 08:18:03.528295 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:03.528325 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:03.528829 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:03.529041 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:03.529044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:03.529323 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:03.529613 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.529617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.530384 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.530385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:03.531164 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.531166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.531282 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.531855 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:03.531875 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:03.533216 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:03.535184 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:03.535189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:03.535191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:03.535428 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:03.535606 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:03.535607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:03.535607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:03.535701 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:03.535916 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:03.535921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:03.536036 (+ 115us) client_negotiation.cc:769] Sending connection context
0503 08:18:03.536132 (+ 96us) client_negotiation.cc:240] Negotiation successful
0503 08:18:03.536265 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20240503 08:18:03.536964 23827 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45221
I20240503 08:18:03.537053 23827 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:03.537240 23827 heartbeater.cc:511] Master 127.6.68.254:45221 requested a full tablet report, sending...
I20240503 08:18:03.537664 23502 ts_manager.cc:194] Registered new tserver with Master: 8c4f22276abb4a83b005e3c2cd020be5 (127.6.68.194:40619)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
I20240503 08:18:03.538132 23502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:33463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:18:03.552595 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45221
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:03.631263 23832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:03.631448 23832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:03.631471 23832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:03.632792 23832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:03.632843 23832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:03.632871 23832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:18:03.634485 23832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:45221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:03.634713 23832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:03.634897 23832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:03.639744 23840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:03.640102 23837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:03.640630 23838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:03.641286 23832 server_base.cc:1034] running on GCE node
I20240503 08:18:03.641449 23832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:03.641639 23832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:03.642766 23832 hybrid_clock.cc:648] HybridClock initialized: now 1714724283642749 us; error 37 us; skew 500 ppm
May 03 08:18:03 dist-test-slave-3g67 krb5kdc[23457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:03.645112 23832 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:18:03.645473 23832 webserver.cc:460] Webserver started at http://127.6.68.195:43453/ using document root <none> and password file <none>
I20240503 08:18:03.645624 23832 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:03.645660 23832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:03.645740 23832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:03.646718 23832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "ec52bb92bbc84624a939acabf876e6d6"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "c5612fd71368d7e81c60c641049d98f2"
server_key_iv: "2c251b802c7bda6d319d6f7dfae4a9fd"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.646984 23832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "ec52bb92bbc84624a939acabf876e6d6"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "c5612fd71368d7e81c60c641049d98f2"
server_key_iv: "2c251b802c7bda6d319d6f7dfae4a9fd"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.648268 23832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:18:03.648911 23847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:03.649071 23832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:03.649138 23832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "ec52bb92bbc84624a939acabf876e6d6"
format_stamp: "Formatted at 2024-05-03 08:18:03 on dist-test-slave-3g67"
server_key: "c5612fd71368d7e81c60c641049d98f2"
server_key_iv: "2c251b802c7bda6d319d6f7dfae4a9fd"
server_key_version: "encryptionkey@0"
I20240503 08:18:03.649219 23832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:03.658733 23832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:03.660452 23832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:03.660564 23832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:03.660753 23832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:03.661026 23832 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:03.661058 23832 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:03.661082 23832 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:03.661129 23832 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:03.666457 23832 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:32783
I20240503 08:18:03.666519 23960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:32783 every 8 connection(s)
I20240503 08:18:03.666800 23832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
May 03 08:18:03 dist-test-slave-3g67 krb5kdc[23457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:03.675782 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.667501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:38745 (local address 127.6.68.254:45221)
0503 08:18:03.667582 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:03.667585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:03.667831 (+ 246us) server_negotiation.cc:407] Connection header received
0503 08:18:03.668481 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:03.668483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:03.668497 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:03.668568 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:03.669559 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.670223 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.671262 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.671436 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.673301 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:03.673305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:03.673305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:03.673312 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:03.674496 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:03.674869 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:03.674870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:03.674870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:03.674897 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:03.675072 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:03.675073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:03.675073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:03.675189 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:03.675259 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:03.675533 (+ 274us) server_negotiation.cc:299] Negotiation successful
0503 08:18:03.675663 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240503 08:18:03.675802 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.667456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:45221 (local address 127.6.68.195:38745)
0503 08:18:03.667725 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:03.667750 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:03.668360 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:03.668645 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:03.668649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:03.669055 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:03.669455 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.669460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.670309 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.670310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:03.671180 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.671183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.671330 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.671895 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:03.671909 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:03.673208 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:03.674589 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:03.674595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:03.674598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:03.674798 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:03.674954 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:03.674955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:03.674955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:03.675013 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:03.675279 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:03.675285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:03.675400 (+ 115us) client_negotiation.cc:769] Sending connection context
0503 08:18:03.675511 (+ 111us) client_negotiation.cc:240] Negotiation successful
0503 08:18:03.675647 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":84,"threads_started":1}
I20240503 08:18:03.676332 23961 heartbeater.cc:347] Connected to a master server at 127.6.68.254:45221
I20240503 08:18:03.676419 23961 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:03.676590 23961 heartbeater.cc:511] Master 127.6.68.254:45221 requested a full tablet report, sending...
I20240503 08:18:03.676824 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 23832
I20240503 08:18:03.676927 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:18:03.676964 23502 ts_manager.cc:194] Registered new tserver with Master: ec52bb92bbc84624a939acabf876e6d6 (127.6.68.195:32783)
I20240503 08:18:03.677094 6419 external_mini_cluster.cc:1428] Setting key ef4b05fd3942fdc2364aec6b2eb7b2d8
I20240503 08:18:03.677441 23502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:38745
I20240503 08:18:03.677847 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:18:03.683888 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.679011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.6.68.254:45221)
0503 08:18:03.679066 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:03.679067 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:03.679160 (+ 93us) server_negotiation.cc:407] Connection header received
0503 08:18:03.679195 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:03.679196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:03.679203 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:03.679251 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:03.679900 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.680346 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.680839 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.680934 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.681837 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:03.681840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:03.681840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:03.681846 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:03.683142 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:03.683374 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:03.683375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:03.683375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:03.683390 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:03.683482 (+ 92us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:03.683482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:03.683483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:03.683578 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:03.683691 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:03.683760 (+ 69us) server_negotiation.cc:299] Negotiation successful
0503 08:18:03.683823 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:18:03.685348 23502 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:18:03.686758 23502 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:18:03.691877 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.688683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:40619 (local address 127.0.0.1:54740)
0503 08:18:03.689087 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:03.689096 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:03.689135 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:03.689368 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:03.689369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:03.689371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:03.689539 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.689540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.690528 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.690529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:03.691276 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.691277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.691351 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.691388 (+ 37us) client_negotiation.cc:769] Sending connection context
0503 08:18:03.691441 (+ 53us) client_negotiation.cc:240] Negotiation successful
0503 08:18:03.691479 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":61,"threads_started":1}
I20240503 08:18:03.691984 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.688681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:32783 (local address 127.0.0.1:56456)
0503 08:18:03.689036 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:03.689045 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:03.689088 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:03.689510 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:03.689511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:03.689513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:03.689719 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.689721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.690979 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.690980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:03.691781 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.691783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.691852 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.691855 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:18:03.691887 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:18:03.691921 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":61,"threads_started":1}
I20240503 08:18:03.692330 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.689063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.6.68.194:40619)
0503 08:18:03.689212 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:03.689219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:03.689228 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:03.689263 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:03.689265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:03.689280 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:03.689362 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:03.689629 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.690447 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.691388 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.691913 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.692176 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:03.692207 (+ 31us) server_negotiation.cc:299] Negotiation successful
0503 08:18:03.692251 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20240503 08:18:03.692615 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.688906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.6.68.193:33591)
0503 08:18:03.689061 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:03.689069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:03.689082 (+ 13us) server_negotiation.cc:407] Connection header received
0503 08:18:03.689123 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:03.689125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:03.689143 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:03.689235 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:03.689614 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.690782 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.691823 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.692287 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.692315 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:03.692505 (+ 190us) server_negotiation.cc:299] Negotiation successful
0503 08:18:03.692555 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20240503 08:18:03.692864 23761 tablet_service.cc:1467] Processing CreateTablet for tablet f4a7a60741b1463aaa87be02af2211bc (DEFAULT_TABLE table=test-table [id=9667d6d794d649d68e6b6ba7f9df15a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:03.693115 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a7a60741b1463aaa87be02af2211bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:03.695315 23975 tablet_bootstrap.cc:492] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: Bootstrap starting.
I20240503 08:18:03.692515 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.688680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:33591 (local address 127.0.0.1:38274)
0503 08:18:03.688845 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:03.688869 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:03.688935 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:03.689250 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:03.689253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:03.689254 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:03.689483 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.689500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.690863 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.690864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:03.691703 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.691705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.692339 (+ 634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.692342 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:18:03.692406 (+ 64us) client_negotiation.cc:240] Negotiation successful
0503 08:18:03.692454 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
I20240503 08:18:03.695680 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.688922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.6.68.195:32783)
0503 08:18:03.689373 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:03.689380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:03.689389 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:03.689423 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:03.689424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:03.689438 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:03.689508 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:03.689807 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.690905 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.692126 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.695453 (+ 3327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.695483 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:03.695511 (+ 28us) server_negotiation.cc:299] Negotiation successful
0503 08:18:03.695566 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":110,"threads_started":1}
I20240503 08:18:03.695971 23975 tablet_bootstrap.cc:654] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:03.696265 23975 log.cc:824] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:03.696230 23895 tablet_service.cc:1467] Processing CreateTablet for tablet f4a7a60741b1463aaa87be02af2211bc (DEFAULT_TABLE table=test-table [id=9667d6d794d649d68e6b6ba7f9df15a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:03.696509 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a7a60741b1463aaa87be02af2211bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:03.696724 23627 tablet_service.cc:1467] Processing CreateTablet for tablet f4a7a60741b1463aaa87be02af2211bc (DEFAULT_TABLE table=test-table [id=9667d6d794d649d68e6b6ba7f9df15a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:03.696964 23627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a7a60741b1463aaa87be02af2211bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:03.698768 23977 tablet_bootstrap.cc:492] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6: Bootstrap starting.
I20240503 08:18:03.699283 23975 tablet_bootstrap.cc:492] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: No bootstrap required, opened a new log
I20240503 08:18:03.699306 23978 tablet_bootstrap.cc:492] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956: Bootstrap starting.
I20240503 08:18:03.699375 23977 tablet_bootstrap.cc:654] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:03.699381 23975 ts_tablet_manager.cc:1367] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:18:03.699726 23977 log.cc:824] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:03.699903 23978 tablet_bootstrap.cc:654] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:03.700261 23978 log.cc:824] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:03.700419 23977 tablet_bootstrap.cc:492] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6: No bootstrap required, opened a new log
I20240503 08:18:03.700552 23977 ts_tablet_manager.cc:1367] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:18:03.701049 23978 tablet_bootstrap.cc:492] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956: No bootstrap required, opened a new log
I20240503 08:18:03.701138 23978 ts_tablet_manager.cc:1367] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:18:03.701951 23975 raft_consensus.cc:348] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec52bb92bbc84624a939acabf876e6d6" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 32783 } } peers { permanent_uuid: "b1ed3eec128a47b6978d93a8d2568956" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33591 } } peers { permanent_uuid: "8c4f22276abb4a83b005e3c2cd020be5" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40619 } }
I20240503 08:18:03.702147 23975 raft_consensus.cc:374] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:03.702173 23975 raft_consensus.cc:729] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4f22276abb4a83b005e3c2cd020be5, State: Initialized, Role: FOLLOWER
I20240503 08:18:03.702312 23975 consensus_queue.cc:260] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec52bb92bbc84624a939acabf876e6d6" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 32783 } } peers { permanent_uuid: "b1ed3eec128a47b6978d93a8d2568956" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33591 } } peers { permanent_uuid: "8c4f22276abb4a83b005e3c2cd020be5" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40619 } }
I20240503 08:18:03.702327 23977 raft_consensus.cc:348] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec52bb92bbc84624a939acabf876e6d6" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 32783 } } peers { permanent_uuid: "b1ed3eec128a47b6978d93a8d2568956" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33591 } } peers { permanent_uuid: "8c4f22276abb4a83b005e3c2cd020be5" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40619 } }
I20240503 08:18:03.702435 23977 raft_consensus.cc:374] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:03.702491 23977 raft_consensus.cc:729] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec52bb92bbc84624a939acabf876e6d6, State: Initialized, Role: FOLLOWER
I20240503 08:18:03.702562 23975 ts_tablet_manager.cc:1398] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:18:03.702590 23977 consensus_queue.cc:260] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec52bb92bbc84624a939acabf876e6d6" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 32783 } } peers { permanent_uuid: "b1ed3eec128a47b6978d93a8d2568956" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33591 } } peers { permanent_uuid: "8c4f22276abb4a83b005e3c2cd020be5" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40619 } }
I20240503 08:18:03.702731 23827 heartbeater.cc:503] Master 127.6.68.254:45221 was elected leader, sending a full tablet report...
I20240503 08:18:03.702953 23961 heartbeater.cc:503] Master 127.6.68.254:45221 was elected leader, sending a full tablet report...
I20240503 08:18:03.702901 23977 ts_tablet_manager.cc:1398] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:18:03.703763 23978 raft_consensus.cc:348] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec52bb92bbc84624a939acabf876e6d6" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 32783 } } peers { permanent_uuid: "b1ed3eec128a47b6978d93a8d2568956" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33591 } } peers { permanent_uuid: "8c4f22276abb4a83b005e3c2cd020be5" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40619 } }
I20240503 08:18:03.703933 23978 raft_consensus.cc:374] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:03.703974 23978 raft_consensus.cc:729] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ed3eec128a47b6978d93a8d2568956, State: Initialized, Role: FOLLOWER
I20240503 08:18:03.704062 23978 consensus_queue.cc:260] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec52bb92bbc84624a939acabf876e6d6" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 32783 } } peers { permanent_uuid: "b1ed3eec128a47b6978d93a8d2568956" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33591 } } peers { permanent_uuid: "8c4f22276abb4a83b005e3c2cd020be5" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40619 } }
I20240503 08:18:03.704272 23978 ts_tablet_manager.cc:1398] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:18:03.704314 23693 heartbeater.cc:503] Master 127.6.68.254:45221 was elected leader, sending a full tablet report...
W20240503 08:18:03.777940 23828 tablet.cc:2369] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:18:03.829811 23983 raft_consensus.cc:482] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:18:03.829946 23983 raft_consensus.cc:504] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec52bb92bbc84624a939acabf876e6d6" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 32783 } } peers { permanent_uuid: "b1ed3eec128a47b6978d93a8d2568956" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33591 } } peers { permanent_uuid: "8c4f22276abb4a83b005e3c2cd020be5" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40619 } }
I20240503 08:18:03.830248 23983 leader_election.cc:288] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec52bb92bbc84624a939acabf876e6d6 (127.6.68.195:32783), 8c4f22276abb4a83b005e3c2cd020be5 (127.6.68.194:40619)
I20240503 08:18:03.832322 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.830440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:32783 (local address 127.6.68.193:38137)
0503 08:18:03.830521 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:03.830534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:03.830581 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:03.830727 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:03.830728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:03.830730 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:03.830923 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.830925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.831554 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.831555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:03.832114 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.832115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.832181 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.832186 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:18:03.832217 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:18:03.832251 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20240503 08:18:03.832677 23984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.830471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:40619 (local address 127.6.68.193:34273)
0503 08:18:03.830671 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:03.830680 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:03.830745 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:03.830879 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:03.830880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:03.830881 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:03.831097 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.831099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.831785 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.831786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:03.832442 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:03.832445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.832549 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.832553 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:18:03.832584 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:18:03.832620 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":125,"threads_started":1}
I20240503 08:18:03.832770 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.830503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:38137 (local address 127.6.68.195:32783)
0503 08:18:03.830584 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:03.830586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:03.830595 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:03.830651 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:03.830652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:03.830660 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:03.830714 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:03.830998 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.831475 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.832195 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.832650 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.832657 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:03.832679 (+ 22us) server_negotiation.cc:299] Negotiation successful
0503 08:18:03.832708 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240503 08:18:03.833062 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a7a60741b1463aaa87be02af2211bc" candidate_uuid: "b1ed3eec128a47b6978d93a8d2568956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec52bb92bbc84624a939acabf876e6d6" is_pre_election: true
I20240503 08:18:03.833132 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.830543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:34273 (local address 127.6.68.194:40619)
0503 08:18:03.830592 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:03.830594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:03.830759 (+ 165us) server_negotiation.cc:407] Connection header received
0503 08:18:03.830796 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:03.830796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:03.830805 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:03.830864 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:03.831176 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.831688 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.832576 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.833025 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.833033 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:03.833054 (+ 21us) server_negotiation.cc:299] Negotiation successful
0503 08:18:03.833087 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240503 08:18:03.833225 23915 raft_consensus.cc:2454] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ed3eec128a47b6978d93a8d2568956 in term 0.
I20240503 08:18:03.833400 23781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a7a60741b1463aaa87be02af2211bc" candidate_uuid: "b1ed3eec128a47b6978d93a8d2568956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c4f22276abb4a83b005e3c2cd020be5" is_pre_election: true
I20240503 08:18:03.833448 23583 leader_election.cc:302] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1ed3eec128a47b6978d93a8d2568956, ec52bb92bbc84624a939acabf876e6d6; no voters:
I20240503 08:18:03.833559 23781 raft_consensus.cc:2454] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ed3eec128a47b6978d93a8d2568956 in term 0.
I20240503 08:18:03.833592 23983 raft_consensus.cc:2781] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:18:03.833693 23983 raft_consensus.cc:482] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:18:03.833732 23983 raft_consensus.cc:3037] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:03.834385 23983 raft_consensus.cc:504] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec52bb92bbc84624a939acabf876e6d6" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 32783 } } peers { permanent_uuid: "b1ed3eec128a47b6978d93a8d2568956" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33591 } } peers { permanent_uuid: "8c4f22276abb4a83b005e3c2cd020be5" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40619 } }
I20240503 08:18:03.834502 23983 leader_election.cc:288] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [CANDIDATE]: Term 1 election: Requested vote from peers ec52bb92bbc84624a939acabf876e6d6 (127.6.68.195:32783), 8c4f22276abb4a83b005e3c2cd020be5 (127.6.68.194:40619)
I20240503 08:18:03.834693 23781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a7a60741b1463aaa87be02af2211bc" candidate_uuid: "b1ed3eec128a47b6978d93a8d2568956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c4f22276abb4a83b005e3c2cd020be5"
I20240503 08:18:03.834693 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a7a60741b1463aaa87be02af2211bc" candidate_uuid: "b1ed3eec128a47b6978d93a8d2568956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec52bb92bbc84624a939acabf876e6d6"
I20240503 08:18:03.834774 23915 raft_consensus.cc:3037] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:03.834774 23781 raft_consensus.cc:3037] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:03.835703 23915 raft_consensus.cc:2454] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ed3eec128a47b6978d93a8d2568956 in term 1.
I20240503 08:18:03.835716 23781 raft_consensus.cc:2454] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ed3eec128a47b6978d93a8d2568956 in term 1.
I20240503 08:18:03.835897 23582 leader_election.cc:302] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c4f22276abb4a83b005e3c2cd020be5, b1ed3eec128a47b6978d93a8d2568956; no voters:
I20240503 08:18:03.836009 23983 raft_consensus.cc:2781] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:03.836166 23983 raft_consensus.cc:686] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 1 LEADER]: Becoming Leader. State: Replica: b1ed3eec128a47b6978d93a8d2568956, State: Running, Role: LEADER
I20240503 08:18:03.836252 23983 consensus_queue.cc:237] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec52bb92bbc84624a939acabf876e6d6" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 32783 } } peers { permanent_uuid: "b1ed3eec128a47b6978d93a8d2568956" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33591 } } peers { permanent_uuid: "8c4f22276abb4a83b005e3c2cd020be5" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40619 } }
I20240503 08:18:03.837015 23501 catalog_manager.cc:5484] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ed3eec128a47b6978d93a8d2568956 (127.6.68.193). New cstate: current_term: 1 leader_uuid: "b1ed3eec128a47b6978d93a8d2568956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec52bb92bbc84624a939acabf876e6d6" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ed3eec128a47b6978d93a8d2568956" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 33591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c4f22276abb4a83b005e3c2cd020be5" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40619 } health_report { overall_health: UNKNOWN } } }
W20240503 08:18:03.870687 23694 tablet.cc:2369] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:18:03.873461 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.871413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.6.68.193:33591)
0503 08:18:03.871474 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:03.871476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:03.871484 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:18:03.871520 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:03.871520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:03.871536 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:03.871603 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:03.871876 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.872354 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:03.872944 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:03.873103 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:03.873141 (+ 38us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:03.873223 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:03.873297 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:03.873355 (+ 58us) server_negotiation.cc:299] Negotiation successful
0503 08:18:03.873379 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240503 08:18:03.875137 23915 raft_consensus.cc:1261] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6 [term 1 FOLLOWER]: Refusing update from remote peer b1ed3eec128a47b6978d93a8d2568956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:03.875137 23781 raft_consensus.cc:1261] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5 [term 1 FOLLOWER]: Refusing update from remote peer b1ed3eec128a47b6978d93a8d2568956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:03.875483 23985 consensus_queue.cc:1035] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec52bb92bbc84624a939acabf876e6d6" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:03.875634 23983 consensus_queue.cc:1035] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c4f22276abb4a83b005e3c2cd020be5" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 40619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:03.878299 23988 mvcc.cc:205] Tried to move back new op lower bound from 7023510666750066688 to 7023510666593886208. Current Snapshot: MvccSnapshot[applied={T|T < 7023510666750066688}]
I20240503 08:18:03.878757 23990 mvcc.cc:205] Tried to move back new op lower bound from 7023510666750066688 to 7023510666593886208. Current Snapshot: MvccSnapshot[applied={T|T < 7023510666750066688 or (T in {7023510666750066688})}]
I20240503 08:18:03.880363 23500 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58380:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:03.880447 23500 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58380:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:03.881173 23500 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0f951be647b4fa683163cb8c54de326: Sending DeleteTablet for 3 replicas of tablet f4a7a60741b1463aaa87be02af2211bc
I20240503 08:18:03.881460 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a7a60741b1463aaa87be02af2211bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:03 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:56456
I20240503 08:18:03.881536 23627 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a7a60741b1463aaa87be02af2211bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:03 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:38274
I20240503 08:18:03.881744 23998 tablet_replica.cc:330] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956: stopping tablet replica
I20240503 08:18:03.881817 23998 raft_consensus.cc:2229] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:18:03.881878 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a7a60741b1463aaa87be02af2211bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:03 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:54740
I20240503 08:18:03.881927 23998 raft_consensus.cc:2258] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:03.882098 23999 tablet_replica.cc:330] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: stopping tablet replica
I20240503 08:18:03.882176 23999 raft_consensus.cc:2229] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:18:03.882252 23999 raft_consensus.cc:2258] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:03.882305 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 23564
I20240503 08:18:03.882373 23998 ts_tablet_manager.cc:1875] T f4a7a60741b1463aaa87be02af2211bc P b1ed3eec128a47b6978d93a8d2568956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:03.883191 23999 ts_tablet_manager.cc:1875] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:03.884747 23999 ts_tablet_manager.cc:1888] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:18:03.884795 23999 log.cc:1196] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/f4a7a60741b1463aaa87be02af2211bc
I20240503 08:18:03.885020 23999 ts_tablet_manager.cc:1909] T f4a7a60741b1463aaa87be02af2211bc P 8c4f22276abb4a83b005e3c2cd020be5: Deleting consensus metadata
I20240503 08:18:03.885358 23488 catalog_manager.cc:4830] TS 8c4f22276abb4a83b005e3c2cd020be5 (127.6.68.194:40619): tablet f4a7a60741b1463aaa87be02af2211bc (table test-table [id=9667d6d794d649d68e6b6ba7f9df15a3]) successfully deleted
W20240503 08:18:03.887995 23486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:33591 (error 108)
I20240503 08:18:03.888159 23997 tablet_replica.cc:330] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6: stopping tablet replica
I20240503 08:18:03.888244 23997 raft_consensus.cc:2229] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:18:03.888249 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 23698
I20240503 08:18:03.888797 23997 raft_consensus.cc:2258] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:03.889144 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:03.889022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:33591 (local address 127.0.0.1:38286)
0503 08:18:03.889074 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:03.889107 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20240503 08:18:03.889250 23997 ts_tablet_manager.cc:1875] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:18:03.889242 23486 catalog_manager.cc:4573] TS b1ed3eec128a47b6978d93a8d2568956 (127.6.68.193:33591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4a7a60741b1463aaa87be02af2211bc: Network error: Client connection negotiation failed: client connection to 127.6.68.193:33591: connect: Connection refused (error 111)
I20240503 08:18:03.890575 23997 ts_tablet_manager.cc:1888] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:18:03.890650 23997 log.cc:1196] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/f4a7a60741b1463aaa87be02af2211bc
I20240503 08:18:03.890928 23997 ts_tablet_manager.cc:1909] T f4a7a60741b1463aaa87be02af2211bc P ec52bb92bbc84624a939acabf876e6d6: Deleting consensus metadata
I20240503 08:18:03.891360 23489 catalog_manager.cc:4830] TS ec52bb92bbc84624a939acabf876e6d6 (127.6.68.195:32783): tablet f4a7a60741b1463aaa87be02af2211bc (table test-table [id=9667d6d794d649d68e6b6ba7f9df15a3]) successfully deleted
I20240503 08:18:03.894006 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 23832
I20240503 08:18:03.898448 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 23470
2024-05-03T08:18:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1942 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2024-05-03T08:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:03Z Disabled control of system clock
I20240503 08:18:03.924554 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:43185
--webserver_interface=127.6.68.254
--webserver_port=0
--builtin_ntp_servers=127.6.68.212:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240503 08:18:04.002414 24004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:04.002604 24004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:04.002624 24004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:04.003870 24004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:04.003904 24004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:04.003916 24004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:04.003928 24004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:04.003939 24004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:04.005290 24004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:43185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:43185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:04.005470 24004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:04.005676 24004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:04.010229 24010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:04.010726 24004 server_base.cc:1034] running on GCE node
W20240503 08:18:04.011324 24009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:04.011843 24012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:04.012077 24004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:04.012820 24004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:04.014089 24004 hybrid_clock.cc:648] HybridClock initialized: now 1714724284014048 us; error 104 us; skew 500 ppm
I20240503 08:18:04.015234 24004 webserver.cc:460] Webserver started at http://127.6.68.254:43537/ using document root <none> and password file <none>
I20240503 08:18:04.015451 24004 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:04.015501 24004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:04.015610 24004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:04.016644 24004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "4a6ae12507154fadab76efd77428c40b"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "31a96b2225d623aa0d5f81e8d79cb8b0"
server_key_iv: "df2aab9676adebbe7be7754638854465"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.016957 24004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "4a6ae12507154fadab76efd77428c40b"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "31a96b2225d623aa0d5f81e8d79cb8b0"
server_key_iv: "df2aab9676adebbe7be7754638854465"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.018115 24004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:18:04.018811 24018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:04.018962 24004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:04.019037 24004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "4a6ae12507154fadab76efd77428c40b"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "31a96b2225d623aa0d5f81e8d79cb8b0"
server_key_iv: "df2aab9676adebbe7be7754638854465"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.019099 24004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:04.044096 24004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:04.044371 24004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:04.044494 24004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:04.048202 24004 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:43185
I20240503 08:18:04.048236 24070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:43185 every 8 connection(s)
I20240503 08:18:04.048540 24004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:04.049116 24071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:04.051559 24071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b: Bootstrap starting.
I20240503 08:18:04.052109 24071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:04.052356 24071 log.cc:824] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:04.053009 24071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b: No bootstrap required, opened a new log
I20240503 08:18:04.054420 24071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6ae12507154fadab76efd77428c40b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43185 } }
I20240503 08:18:04.054543 24071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:04.054561 24071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6ae12507154fadab76efd77428c40b, State: Initialized, Role: FOLLOWER
I20240503 08:18:04.054687 24071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6ae12507154fadab76efd77428c40b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43185 } }
I20240503 08:18:04.054749 24071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:04.054771 24071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:04.054841 24071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:04.055343 24071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6ae12507154fadab76efd77428c40b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43185 } }
I20240503 08:18:04.055433 24071 leader_election.cc:302] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a6ae12507154fadab76efd77428c40b; no voters:
I20240503 08:18:04.055584 24071 leader_election.cc:288] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:04.055666 24074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:04.055807 24074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [term 1 LEADER]: Becoming Leader. State: Replica: 4a6ae12507154fadab76efd77428c40b, State: Running, Role: LEADER
I20240503 08:18:04.055830 24071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:04.055886 24074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6ae12507154fadab76efd77428c40b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43185 } }
I20240503 08:18:04.056190 24075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a6ae12507154fadab76efd77428c40b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6ae12507154fadab76efd77428c40b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43185 } } }
I20240503 08:18:04.056200 24076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a6ae12507154fadab76efd77428c40b. Latest consensus state: current_term: 1 leader_uuid: "4a6ae12507154fadab76efd77428c40b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6ae12507154fadab76efd77428c40b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 43185 } } }
I20240503 08:18:04.056351 24075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:04.056365 24076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:04.056790 24079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:04.057312 24079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:04.058195 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 24004
I20240503 08:18:04.058310 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:04.058498 6419 external_mini_cluster.cc:1428] Setting key 1b8341080ffc09802775abc2fdb6929a
I20240503 08:18:04.058712 24079 catalog_manager.cc:1347] Generated new cluster ID: 6e8e2b351440449f88f8945eaa54e094
I20240503 08:18:04.058794 24079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:04.063202 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.058919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.6.68.254:43185)
0503 08:18:04.059219 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:04.059226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:04.059246 (+ 20us) server_negotiation.cc:407] Connection header received
0503 08:18:04.059334 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:04.059339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:04.059389 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:04.059529 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:04.060009 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.061172 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.061964 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.062173 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.062608 (+ 435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:04.062614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:04.062615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:04.062802 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:04.062831 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:04.062838 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:04.062904 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:04.062993 (+ 89us) server_negotiation.cc:299] Negotiation successful
0503 08:18:04.063040 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20240503 08:18:04.078692 24079 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:04.079286 24079 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:04.087010 24079 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b: Generated new TSK 0
I20240503 08:18:04.087193 24079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:04.092972 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43185
--builtin_ntp_servers=127.6.68.212:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240503 08:18:04.172518 24095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:04.172693 24095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:04.172713 24095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:04.174014 24095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:04.174059 24095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:04.174084 24095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:18:04.175472 24095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:04.175683 24095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:04.175930 24095 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:04.181337 24100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:04.181815 24101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:04.182526 24103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:04.183307 24095 server_base.cc:1034] running on GCE node
I20240503 08:18:04.183602 24095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:04.183904 24095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:04.185060 24095 hybrid_clock.cc:648] HybridClock initialized: now 1714724284185048 us; error 35 us; skew 500 ppm
I20240503 08:18:04.186369 24095 webserver.cc:460] Webserver started at http://127.6.68.193:37467/ using document root <none> and password file <none>
I20240503 08:18:04.186614 24095 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:04.186657 24095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:04.186760 24095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:04.187815 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "86917ac50c834e449f45dac272eecbbe"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "a6d0fa0180d57f026566dedc361cbf2d"
server_key_iv: "7c425e4ec3a779e5cccf5c8fc9afdbd1"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.188117 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "86917ac50c834e449f45dac272eecbbe"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "a6d0fa0180d57f026566dedc361cbf2d"
server_key_iv: "7c425e4ec3a779e5cccf5c8fc9afdbd1"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.189280 24095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:04.189968 24109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:04.190138 24095 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:04.190217 24095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "86917ac50c834e449f45dac272eecbbe"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "a6d0fa0180d57f026566dedc361cbf2d"
server_key_iv: "7c425e4ec3a779e5cccf5c8fc9afdbd1"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.190291 24095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:04.205947 24095 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:04.206243 24095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:04.206357 24095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:04.206566 24095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:04.206885 24095 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:04.206931 24095 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:04.206965 24095 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:04.207012 24095 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:04.212757 24095 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:34701
I20240503 08:18:04.212798 24222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:34701 every 8 connection(s)
I20240503 08:18:04.213102 24095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:18:04.217456 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.213814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:56831 (local address 127.6.68.254:43185)
0503 08:18:04.213907 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:04.213909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:04.214003 (+ 94us) server_negotiation.cc:407] Connection header received
0503 08:18:04.214470 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:04.214471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:04.214486 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:04.214538 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:04.214995 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.215479 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.216074 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.216216 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.217164 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:04.217168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:04.217169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:04.217173 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:04.217185 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:04.217188 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:04.217240 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:04.217326 (+ 86us) server_negotiation.cc:299] Negotiation successful
0503 08:18:04.217369 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240503 08:18:04.217497 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.213739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43185 (local address 127.6.68.193:56831)
0503 08:18:04.213902 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:04.213925 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:04.214353 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:04.214579 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:04.214582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:04.214589 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:04.214910 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.214913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.215575 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.215575 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:04.215993 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.215994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.216098 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.217050 (+ 952us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:18:04.217050 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:04.217070 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:18:04.217088 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:04.217252 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:04.217254 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:04.217256 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:18:04.217310 (+ 54us) client_negotiation.cc:240] Negotiation successful
0503 08:18:04.217376 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20240503 08:18:04.217506 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 24095
I20240503 08:18:04.217602 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:18:04.217743 6419 external_mini_cluster.cc:1428] Setting key 8cfad02baaff55284f4cf4f61c369507
I20240503 08:18:04.218307 24223 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43185
I20240503 08:18:04.218389 24223 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:04.218595 24223 heartbeater.cc:511] Master 127.6.68.254:43185 requested a full tablet report, sending...
I20240503 08:18:04.218767 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43185
--builtin_ntp_servers=127.6.68.212:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240503 08:18:04.219125 24035 ts_manager.cc:194] Registered new tserver with Master: 86917ac50c834e449f45dac272eecbbe (127.6.68.193:34701)
I20240503 08:18:04.219816 24035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.193:56831
W20240503 08:18:04.296926 24226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:04.297099 24226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:04.297122 24226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:04.298334 24226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:04.298372 24226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:04.298395 24226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:18:04.299778 24226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:04.299934 24226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:04.300130 24226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:04.304803 24231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:04.305425 24232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:04.305959 24234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:04.306586 24226 server_base.cc:1034] running on GCE node
I20240503 08:18:04.306747 24226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:04.306944 24226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:04.308094 24226 hybrid_clock.cc:648] HybridClock initialized: now 1714724284308076 us; error 32 us; skew 500 ppm
I20240503 08:18:04.309387 24226 webserver.cc:460] Webserver started at http://127.6.68.194:38549/ using document root <none> and password file <none>
I20240503 08:18:04.309578 24226 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:04.309624 24226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:04.309736 24226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:04.310717 24226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "b965530e409343e689d45259f33a9777"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "f025f5eaa966de0d0ece1d7852eaf6d2"
server_key_iv: "537726be4da4be48b62cd6c0bd58e2ff"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.311012 24226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "b965530e409343e689d45259f33a9777"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "f025f5eaa966de0d0ece1d7852eaf6d2"
server_key_iv: "537726be4da4be48b62cd6c0bd58e2ff"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.312219 24226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:18:04.312930 24240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:04.313089 24226 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:04.313165 24226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "b965530e409343e689d45259f33a9777"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "f025f5eaa966de0d0ece1d7852eaf6d2"
server_key_iv: "537726be4da4be48b62cd6c0bd58e2ff"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.313234 24226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:04.330890 24226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:04.331141 24226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:04.331251 24226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:04.331466 24226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:04.331810 24226 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:04.331848 24226 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:04.331880 24226 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:04.331913 24226 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:04.337208 24226 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:36267
I20240503 08:18:04.337252 24353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:36267 every 8 connection(s)
I20240503 08:18:04.337561 24226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:18:04.341957 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.338274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:43467 (local address 127.6.68.254:43185)
0503 08:18:04.338360 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:04.338362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:04.338544 (+ 182us) server_negotiation.cc:407] Connection header received
0503 08:18:04.338959 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:04.338960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:04.338970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:04.339028 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:04.339443 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.339978 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.340546 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.340685 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.341700 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:04.341704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:04.341705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:04.341709 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:04.341724 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:04.341728 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:04.341778 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:04.341860 (+ 82us) server_negotiation.cc:299] Negotiation successful
0503 08:18:04.341899 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20240503 08:18:04.342068 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.338237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43185 (local address 127.6.68.194:43467)
0503 08:18:04.338415 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:04.338438 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:04.338872 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:04.339079 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:04.339082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:04.339089 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:04.339376 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.339379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.340078 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.340079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:04.340445 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.340446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.340557 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.341561 (+ 1004us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:18:04.341562 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:04.341581 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:18:04.341599 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:04.341790 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:04.341793 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:04.341794 (+ 1us) client_negotiation.cc:769] Sending connection context
0503 08:18:04.341853 (+ 59us) client_negotiation.cc:240] Negotiation successful
0503 08:18:04.341925 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20240503 08:18:04.342576 24354 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43185
I20240503 08:18:04.342656 24354 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:04.342846 24354 heartbeater.cc:511] Master 127.6.68.254:43185 requested a full tablet report, sending...
I20240503 08:18:04.343241 24035 ts_manager.cc:194] Registered new tserver with Master: b965530e409343e689d45259f33a9777 (127.6.68.194:36267)
I20240503 08:18:04.343443 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 24226
I20240503 08:18:04.343544 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:18:04.343731 6419 external_mini_cluster.cc:1428] Setting key da0fdfc0834cf42724e4375278c0dcf8
I20240503 08:18:04.343832 24035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.194:43467
I20240503 08:18:04.344694 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43185
--builtin_ntp_servers=127.6.68.212:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240503 08:18:04.421401 24357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:04.421572 24357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:04.421593 24357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:04.422905 24357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:04.422946 24357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:04.422971 24357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:18:04.424401 24357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:04.424594 24357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:04.424825 24357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:04.429457 24365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:04.429876 24362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:04.430440 24363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:04.431071 24357 server_base.cc:1034] running on GCE node
I20240503 08:18:04.431226 24357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:04.431423 24357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:04.432569 24357 hybrid_clock.cc:648] HybridClock initialized: now 1714724284432538 us; error 39 us; skew 500 ppm
I20240503 08:18:04.433849 24357 webserver.cc:460] Webserver started at http://127.6.68.195:36949/ using document root <none> and password file <none>
I20240503 08:18:04.434038 24357 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:04.434084 24357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:04.434196 24357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:04.435149 24357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "a17be6e80d174cbeab8898f04b9ad40e"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "e894c7d0a55a65b2d28cfdeabfcec1f5"
server_key_iv: "86b95aedfc876f4f0839fd7d165a246b"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.435444 24357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "a17be6e80d174cbeab8898f04b9ad40e"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "e894c7d0a55a65b2d28cfdeabfcec1f5"
server_key_iv: "86b95aedfc876f4f0839fd7d165a246b"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.436609 24357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:18:04.437273 24371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:04.437434 24357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:04.437520 24357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "a17be6e80d174cbeab8898f04b9ad40e"
format_stamp: "Formatted at 2024-05-03 08:18:04 on dist-test-slave-3g67"
server_key: "e894c7d0a55a65b2d28cfdeabfcec1f5"
server_key_iv: "86b95aedfc876f4f0839fd7d165a246b"
server_key_version: "encryptionkey@0"
I20240503 08:18:04.437582 24357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:04.457520 24357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:04.457768 24357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:04.457880 24357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:04.458076 24357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:04.458369 24357 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:04.458402 24357 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:04.458438 24357 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:04.458467 24357 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:04.463598 24357 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:46349
I20240503 08:18:04.463690 24484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:46349 every 8 connection(s)
I20240503 08:18:04.464011 24357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:18:04.468281 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.464671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:54999 (local address 127.6.68.254:43185)
0503 08:18:04.464716 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:04.464718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:04.464938 (+ 220us) server_negotiation.cc:407] Connection header received
0503 08:18:04.465404 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:04.465405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:04.465417 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:04.465470 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:04.465915 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.466384 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.466920 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.467089 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.468008 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:04.468012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:04.468012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:04.468017 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:04.468033 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:04.468037 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:04.468084 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:04.468175 (+ 91us) server_negotiation.cc:299] Negotiation successful
0503 08:18:04.468206 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:18:04.468492 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.464645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:43185 (local address 127.6.68.195:54999)
0503 08:18:04.464836 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:04.464859 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:04.465303 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:04.465529 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:04.465532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:04.465540 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:04.465826 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.465829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.466475 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.466476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:04.466840 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.466842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.466970 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.467893 (+ 923us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:18:04.467893 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:04.467912 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:18:04.467929 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:04.468091 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:04.468095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:04.468097 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:18:04.468178 (+ 81us) client_negotiation.cc:240] Negotiation successful
0503 08:18:04.468250 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20240503 08:18:04.468984 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 24357
I20240503 08:18:04.469064 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:18:04.469044 24485 heartbeater.cc:347] Connected to a master server at 127.6.68.254:43185
I20240503 08:18:04.469154 24485 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:04.469237 6419 external_mini_cluster.cc:1428] Setting key c2beedfa8f704f98f8a6d7c095e4ebdf
I20240503 08:18:04.469390 24485 heartbeater.cc:511] Master 127.6.68.254:43185 requested a full tablet report, sending...
I20240503 08:18:04.469826 24035 ts_manager.cc:194] Registered new tserver with Master: a17be6e80d174cbeab8898f04b9ad40e (127.6.68.195:46349)
I20240503 08:18:04.470211 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:18:04.470297 24035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.195:54999
I20240503 08:18:04.474205 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.471850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.6.68.254:43185)
0503 08:18:04.471920 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:04.471921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:04.471988 (+ 67us) server_negotiation.cc:407] Connection header received
0503 08:18:04.472046 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:04.472046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:04.472053 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:04.472091 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:04.472431 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.472878 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.473496 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.473613 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.474028 (+ 415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:04.474031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:04.474031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:04.474035 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:04.474046 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:04.474049 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:04.474092 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:04.474131 (+ 39us) server_negotiation.cc:299] Negotiation successful
0503 08:18:04.474152 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20240503 08:18:04.475641 24035 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:18:04.477135 24035 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:18:04.482084 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.479040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:36267 (local address 127.0.0.1:47790)
0503 08:18:04.479176 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:04.479206 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:04.479258 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:04.479731 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:04.479732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:04.479734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:04.479888 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.479890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.481299 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.481300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:04.481876 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.481878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.481923 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.481967 (+ 44us) client_negotiation.cc:769] Sending connection context
0503 08:18:04.482002 (+ 35us) client_negotiation.cc:240] Negotiation successful
0503 08:18:04.482032 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20240503 08:18:04.482399 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.479070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:34701 (local address 127.0.0.1:37614)
0503 08:18:04.479335 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:04.479344 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:04.479419 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:04.479658 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:04.479661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:04.479663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:04.479894 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.479896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.481288 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.481289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:04.482155 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.482156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.482226 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.482227 (+ 1us) client_negotiation.cc:769] Sending connection context
0503 08:18:04.482261 (+ 34us) client_negotiation.cc:240] Negotiation successful
0503 08:18:04.482296 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":86,"threads_started":1}
I20240503 08:18:04.482486 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.479259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:46349 (local address 127.0.0.1:40856)
0503 08:18:04.479520 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:04.479528 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:04.479574 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:04.480186 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:04.480187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:04.480189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:04.480335 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.480336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.481236 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.481237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:04.482170 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.482171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.482224 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.482227 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:18:04.482261 (+ 34us) client_negotiation.cc:240] Negotiation successful
0503 08:18:04.482296 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":63,"threads_started":1}
I20240503 08:18:04.482973 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.479390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.6.68.194:36267)
0503 08:18:04.479553 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:04.479560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:04.479569 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:04.479606 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:04.479608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:04.479652 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:04.479732 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:04.479977 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.481195 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.482311 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.482806 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.482867 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:04.482892 (+ 25us) server_negotiation.cc:299] Negotiation successful
0503 08:18:04.482914 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20240503 08:18:04.483132 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.479168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.6.68.193:34701)
0503 08:18:04.479351 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:04.479359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:04.479418 (+ 59us) server_negotiation.cc:407] Connection header received
0503 08:18:04.479497 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:04.479499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:04.479517 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:04.479976 (+ 459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:04.480054 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.481219 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.482579 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.482934 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.482985 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:04.483027 (+ 42us) server_negotiation.cc:299] Negotiation successful
0503 08:18:04.483048 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20240503 08:18:04.483594 24288 tablet_service.cc:1467] Processing CreateTablet for tablet 16d94438c69d43bcb982c8d5e5ffd068 (DEFAULT_TABLE table=test-table [id=534d0e6a39c94d7097f3a71c0f999c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:04.483594 24157 tablet_service.cc:1467] Processing CreateTablet for tablet 16d94438c69d43bcb982c8d5e5ffd068 (DEFAULT_TABLE table=test-table [id=534d0e6a39c94d7097f3a71c0f999c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:04.483904 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.479594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.6.68.195:46349)
0503 08:18:04.480018 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:04.480025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:04.480034 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:04.480071 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:04.480073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:04.480089 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:04.480396 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:04.480456 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.481162 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.483324 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.483719 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.483774 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:04.483802 (+ 28us) server_negotiation.cc:299] Negotiation successful
0503 08:18:04.483836 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":107,"threads_started":1}
I20240503 08:18:04.483919 24288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d94438c69d43bcb982c8d5e5ffd068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:04.483919 24157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d94438c69d43bcb982c8d5e5ffd068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:04.484396 24419 tablet_service.cc:1467] Processing CreateTablet for tablet 16d94438c69d43bcb982c8d5e5ffd068 (DEFAULT_TABLE table=test-table [id=534d0e6a39c94d7097f3a71c0f999c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:04.484635 24419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d94438c69d43bcb982c8d5e5ffd068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:04.486819 24500 tablet_bootstrap.cc:492] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777: Bootstrap starting.
I20240503 08:18:04.487072 24499 tablet_bootstrap.cc:492] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e: Bootstrap starting.
I20240503 08:18:04.487399 24500 tablet_bootstrap.cc:654] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:04.487730 24499 tablet_bootstrap.cc:654] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:04.487742 24500 log.cc:824] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:04.487785 24501 tablet_bootstrap.cc:492] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe: Bootstrap starting.
I20240503 08:18:04.488035 24499 log.cc:824] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:04.488322 24501 tablet_bootstrap.cc:654] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:04.488503 24500 tablet_bootstrap.cc:492] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777: No bootstrap required, opened a new log
I20240503 08:18:04.488581 24500 ts_tablet_manager.cc:1367] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:18:04.488579 24501 log.cc:824] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:04.488638 24499 tablet_bootstrap.cc:492] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e: No bootstrap required, opened a new log
I20240503 08:18:04.488698 24499 ts_tablet_manager.cc:1367] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:18:04.489133 24501 tablet_bootstrap.cc:492] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe: No bootstrap required, opened a new log
I20240503 08:18:04.489189 24501 ts_tablet_manager.cc:1367] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:04.490896 24500 raft_consensus.cc:348] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b965530e409343e689d45259f33a9777" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36267 } } peers { permanent_uuid: "a17be6e80d174cbeab8898f04b9ad40e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 46349 } } peers { permanent_uuid: "86917ac50c834e449f45dac272eecbbe" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 34701 } }
I20240503 08:18:04.490900 24499 raft_consensus.cc:348] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b965530e409343e689d45259f33a9777" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36267 } } peers { permanent_uuid: "a17be6e80d174cbeab8898f04b9ad40e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 46349 } } peers { permanent_uuid: "86917ac50c834e449f45dac272eecbbe" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 34701 } }
I20240503 08:18:04.491036 24499 raft_consensus.cc:374] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:04.491036 24500 raft_consensus.cc:374] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:04.491065 24499 raft_consensus.cc:729] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17be6e80d174cbeab8898f04b9ad40e, State: Initialized, Role: FOLLOWER
I20240503 08:18:04.491065 24500 raft_consensus.cc:729] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b965530e409343e689d45259f33a9777, State: Initialized, Role: FOLLOWER
I20240503 08:18:04.490998 24501 raft_consensus.cc:348] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b965530e409343e689d45259f33a9777" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36267 } } peers { permanent_uuid: "a17be6e80d174cbeab8898f04b9ad40e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 46349 } } peers { permanent_uuid: "86917ac50c834e449f45dac272eecbbe" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 34701 } }
I20240503 08:18:04.491117 24501 raft_consensus.cc:374] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:04.491142 24501 raft_consensus.cc:729] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86917ac50c834e449f45dac272eecbbe, State: Initialized, Role: FOLLOWER
I20240503 08:18:04.491159 24499 consensus_queue.cc:260] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b965530e409343e689d45259f33a9777" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36267 } } peers { permanent_uuid: "a17be6e80d174cbeab8898f04b9ad40e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 46349 } } peers { permanent_uuid: "86917ac50c834e449f45dac272eecbbe" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 34701 } }
I20240503 08:18:04.491197 24500 consensus_queue.cc:260] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b965530e409343e689d45259f33a9777" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36267 } } peers { permanent_uuid: "a17be6e80d174cbeab8898f04b9ad40e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 46349 } } peers { permanent_uuid: "86917ac50c834e449f45dac272eecbbe" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 34701 } }
I20240503 08:18:04.491240 24501 consensus_queue.cc:260] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b965530e409343e689d45259f33a9777" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36267 } } peers { permanent_uuid: "a17be6e80d174cbeab8898f04b9ad40e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 46349 } } peers { permanent_uuid: "86917ac50c834e449f45dac272eecbbe" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 34701 } }
I20240503 08:18:04.491397 24500 ts_tablet_manager.cc:1398] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:18:04.491397 24499 ts_tablet_manager.cc:1398] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:18:04.491479 24501 ts_tablet_manager.cc:1398] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:18:04.491518 24354 heartbeater.cc:503] Master 127.6.68.254:43185 was elected leader, sending a full tablet report...
I20240503 08:18:04.491639 24223 heartbeater.cc:503] Master 127.6.68.254:43185 was elected leader, sending a full tablet report...
I20240503 08:18:04.491770 24485 heartbeater.cc:503] Master 127.6.68.254:43185 was elected leader, sending a full tablet report...
W20240503 08:18:04.588179 24355 tablet.cc:2369] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:18:04.639410 24506 raft_consensus.cc:482] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:18:04.639562 24506 raft_consensus.cc:504] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b965530e409343e689d45259f33a9777" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36267 } } peers { permanent_uuid: "a17be6e80d174cbeab8898f04b9ad40e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 46349 } } peers { permanent_uuid: "86917ac50c834e449f45dac272eecbbe" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 34701 } }
I20240503 08:18:04.639982 24506 leader_election.cc:288] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a17be6e80d174cbeab8898f04b9ad40e (127.6.68.195:46349), 86917ac50c834e449f45dac272eecbbe (127.6.68.193:34701)
I20240503 08:18:04.642341 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.640147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:34701 (local address 127.6.68.194:41997)
0503 08:18:04.640223 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:04.640235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:04.640285 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:04.640507 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:04.640508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:04.640509 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:04.640752 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.640754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.641566 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.641567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:04.642105 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.642107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.642172 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.642177 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:18:04.642208 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:18:04.642241 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20240503 08:18:04.642522 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.640179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:46349 (local address 127.6.68.194:42515)
0503 08:18:04.640379 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:04.640387 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:04.640447 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:04.640574 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:04.640574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:04.640575 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:04.640817 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.640819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.641518 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.641519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:04.642128 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:04.642129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.642212 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.642215 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:18:04.642249 (+ 34us) client_negotiation.cc:240] Negotiation successful
0503 08:18:04.642282 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
I20240503 08:18:04.642760 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.640326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:41997 (local address 127.6.68.193:34701)
0503 08:18:04.640367 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:04.640369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:04.640376 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:04.640399 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:04.640400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:04.640409 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:04.640462 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:04.640859 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.641445 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.642185 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.642651 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.642659 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:04.642682 (+ 23us) server_negotiation.cc:299] Negotiation successful
0503 08:18:04.642711 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:18:04.642853 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.640259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:42515 (local address 127.6.68.195:46349)
0503 08:18:04.640323 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:04.640325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:04.640453 (+ 128us) server_negotiation.cc:407] Connection header received
0503 08:18:04.640496 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:04.640497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:04.640509 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:04.640558 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:04.640879 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.641430 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.642213 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.642721 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.642729 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:04.642752 (+ 23us) server_negotiation.cc:299] Negotiation successful
0503 08:18:04.642782 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:18:04.643095 24439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d94438c69d43bcb982c8d5e5ffd068" candidate_uuid: "b965530e409343e689d45259f33a9777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17be6e80d174cbeab8898f04b9ad40e" is_pre_election: true
I20240503 08:18:04.643100 24177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d94438c69d43bcb982c8d5e5ffd068" candidate_uuid: "b965530e409343e689d45259f33a9777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86917ac50c834e449f45dac272eecbbe" is_pre_election: true
I20240503 08:18:04.643206 24177 raft_consensus.cc:2454] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b965530e409343e689d45259f33a9777 in term 0.
I20240503 08:18:04.643218 24439 raft_consensus.cc:2454] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b965530e409343e689d45259f33a9777 in term 0.
I20240503 08:18:04.643420 24244 leader_election.cc:302] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17be6e80d174cbeab8898f04b9ad40e, b965530e409343e689d45259f33a9777; no voters:
I20240503 08:18:04.643572 24506 raft_consensus.cc:2781] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:18:04.643646 24506 raft_consensus.cc:482] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:18:04.643679 24506 raft_consensus.cc:3037] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:04.644300 24506 raft_consensus.cc:504] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b965530e409343e689d45259f33a9777" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36267 } } peers { permanent_uuid: "a17be6e80d174cbeab8898f04b9ad40e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 46349 } } peers { permanent_uuid: "86917ac50c834e449f45dac272eecbbe" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 34701 } }
I20240503 08:18:04.644421 24506 leader_election.cc:288] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [CANDIDATE]: Term 1 election: Requested vote from peers a17be6e80d174cbeab8898f04b9ad40e (127.6.68.195:46349), 86917ac50c834e449f45dac272eecbbe (127.6.68.193:34701)
I20240503 08:18:04.644598 24439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d94438c69d43bcb982c8d5e5ffd068" candidate_uuid: "b965530e409343e689d45259f33a9777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17be6e80d174cbeab8898f04b9ad40e"
I20240503 08:18:04.644604 24177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d94438c69d43bcb982c8d5e5ffd068" candidate_uuid: "b965530e409343e689d45259f33a9777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86917ac50c834e449f45dac272eecbbe"
I20240503 08:18:04.644672 24439 raft_consensus.cc:3037] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:04.644672 24177 raft_consensus.cc:3037] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:04.645522 24177 raft_consensus.cc:2454] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b965530e409343e689d45259f33a9777 in term 1.
I20240503 08:18:04.645534 24439 raft_consensus.cc:2454] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b965530e409343e689d45259f33a9777 in term 1.
I20240503 08:18:04.645704 24244 leader_election.cc:302] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17be6e80d174cbeab8898f04b9ad40e, b965530e409343e689d45259f33a9777; no voters:
I20240503 08:18:04.645794 24506 raft_consensus.cc:2781] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:04.645925 24506 raft_consensus.cc:686] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 1 LEADER]: Becoming Leader. State: Replica: b965530e409343e689d45259f33a9777, State: Running, Role: LEADER
I20240503 08:18:04.646014 24506 consensus_queue.cc:237] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b965530e409343e689d45259f33a9777" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36267 } } peers { permanent_uuid: "a17be6e80d174cbeab8898f04b9ad40e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 46349 } } peers { permanent_uuid: "86917ac50c834e449f45dac272eecbbe" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 34701 } }
I20240503 08:18:04.646777 24034 catalog_manager.cc:5484] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 reported cstate change: term changed from 0 to 1, leader changed from <none> to b965530e409343e689d45259f33a9777 (127.6.68.194). New cstate: current_term: 1 leader_uuid: "b965530e409343e689d45259f33a9777" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b965530e409343e689d45259f33a9777" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 36267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17be6e80d174cbeab8898f04b9ad40e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 46349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86917ac50c834e449f45dac272eecbbe" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 34701 } health_report { overall_health: UNKNOWN } } }
I20240503 08:18:04.666363 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.664386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.6.68.194:36267)
0503 08:18:04.664448 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:04.664450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:04.664458 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:18:04.664496 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:04.664497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:04.664515 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:04.664594 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:04.664838 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.665355 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:04.665931 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:04.666072 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:04.666111 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:04.666192 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:04.666255 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:04.666288 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:18:04.666306 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240503 08:18:04.668051 24177 raft_consensus.cc:1261] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe [term 1 FOLLOWER]: Refusing update from remote peer b965530e409343e689d45259f33a9777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:04.668056 24439 raft_consensus.cc:1261] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e [term 1 FOLLOWER]: Refusing update from remote peer b965530e409343e689d45259f33a9777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:04.668331 24506 consensus_queue.cc:1035] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17be6e80d174cbeab8898f04b9ad40e" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:04.668450 24509 consensus_queue.cc:1035] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86917ac50c834e449f45dac272eecbbe" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:04.671197 24514 mvcc.cc:205] Tried to move back new op lower bound from 7023510669997756416 to 7023510669910544384. Current Snapshot: MvccSnapshot[applied={T|T < 7023510669997756416}]
I20240503 08:18:04.673241 24035 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54338:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:04.673350 24035 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54338:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:04.674098 24035 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 4a6ae12507154fadab76efd77428c40b: Sending DeleteTablet for 3 replicas of tablet 16d94438c69d43bcb982c8d5e5ffd068
I20240503 08:18:04.674623 24288 tablet_service.cc:1514] Processing DeleteTablet for tablet 16d94438c69d43bcb982c8d5e5ffd068 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:04 UTC) from {username='slave'} at 127.0.0.1:47790
I20240503 08:18:04.674655 24419 tablet_service.cc:1514] Processing DeleteTablet for tablet 16d94438c69d43bcb982c8d5e5ffd068 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:04 UTC) from {username='slave'} at 127.0.0.1:40856
I20240503 08:18:04.674743 24157 tablet_service.cc:1514] Processing DeleteTablet for tablet 16d94438c69d43bcb982c8d5e5ffd068 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:04 UTC) from {username='slave'} at 127.0.0.1:37614
I20240503 08:18:04.674927 24523 tablet_replica.cc:330] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e: stopping tablet replica
I20240503 08:18:04.674957 24524 tablet_replica.cc:330] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe: stopping tablet replica
I20240503 08:18:04.675032 24524 raft_consensus.cc:2229] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:18:04.675016 24523 raft_consensus.cc:2229] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:18:04.675118 24523 raft_consensus.cc:2258] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:04.675119 24524 raft_consensus.cc:2258] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:04.675477 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 24095
I20240503 08:18:04.675525 24524 ts_tablet_manager.cc:1875] T 16d94438c69d43bcb982c8d5e5ffd068 P 86917ac50c834e449f45dac272eecbbe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:18:04.681865 24019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:34701 (error 108)
I20240503 08:18:04.681954 24522 tablet_replica.cc:330] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777: stopping tablet replica
I20240503 08:18:04.682044 24522 raft_consensus.cc:2229] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:18:04.682148 24522 raft_consensus.cc:2258] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:04.682205 24523 ts_tablet_manager.cc:1875] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:04.682619 24522 ts_tablet_manager.cc:1875] T 16d94438c69d43bcb982c8d5e5ffd068 P b965530e409343e689d45259f33a9777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:04.682708 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 24226
I20240503 08:18:04.684209 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.683752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:34701 (local address 127.0.0.1:37618)
0503 08:18:04.684116 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:04.684163 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
I20240503 08:18:04.684244 24523 ts_tablet_manager.cc:1888] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:18:04.684309 24523 log.cc:1196] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/16d94438c69d43bcb982c8d5e5ffd068
I20240503 08:18:04.684587 24523 ts_tablet_manager.cc:1909] T 16d94438c69d43bcb982c8d5e5ffd068 P a17be6e80d174cbeab8898f04b9ad40e: Deleting consensus metadata
W20240503 08:18:04.684782 24019 catalog_manager.cc:4573] TS 86917ac50c834e449f45dac272eecbbe (127.6.68.193:34701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 16d94438c69d43bcb982c8d5e5ffd068: Network error: Client connection negotiation failed: client connection to 127.6.68.193:34701: connect: Connection refused (error 111)
I20240503 08:18:04.684989 24022 catalog_manager.cc:4830] TS a17be6e80d174cbeab8898f04b9ad40e (127.6.68.195:46349): tablet 16d94438c69d43bcb982c8d5e5ffd068 (table test-table [id=534d0e6a39c94d7097f3a71c0f999c21]) successfully deleted
I20240503 08:18:04.688247 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 24357
I20240503 08:18:04.688382 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:04.688261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:36267 (local address 127.0.0.1:47822)
0503 08:18:04.688319 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:04.688346 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.194:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20240503 08:18:04.692978 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 24004
2024-05-03T08:18:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (794 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:04 dist-test-slave-3g67 krb5kdc[24528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:04 dist-test-slave-3g67 krb5kdc[24528](info): set up 2 sockets
May 03 08:18:04 dist-test-slave-3g67 krb5kdc[24528](info): commencing operation
krb5kdc: starting...
W20240503 08:18:05.647845 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:05 dist-test-slave-3g67 krb5kdc[24528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:18:05.770957 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:35113
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:35113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:05.847379 24541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:05.847582 24541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:05.847616 24541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:05.848923 24541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:05.848968 24541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:05.848989 24541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:05.849009 24541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:05.849035 24541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:05.850620 24541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:35113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:35113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:05.850842 24541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:05.851017 24541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:05.855479 24547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:05.856035 24549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:05.856635 24546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:05.857220 24541 server_base.cc:1034] running on GCE node
I20240503 08:18:05.857548 24541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:05.857793 24541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:05.858952 24541 hybrid_clock.cc:648] HybridClock initialized: now 1714724285858932 us; error 31 us; skew 500 ppm
May 03 08:18:05 dist-test-slave-3g67 krb5kdc[24528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724285, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:05.861479 24541 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:18:05.861891 24541 webserver.cc:460] Webserver started at http://127.6.68.254:45085/ using document root <none> and password file <none>
I20240503 08:18:05.862062 24541 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:05.862105 24541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:05.862226 24541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:05.863232 24541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "bd4b6cbe1c564b29b09c7390f066a461"
format_stamp: "Formatted at 2024-05-03 08:18:05 on dist-test-slave-3g67"
server_key: "126e740e3a04e6a07b915441c6792040"
server_key_iv: "2c4259d1645ae3fd6c043172ec0c3772"
server_key_version: "encryptionkey@0"
I20240503 08:18:05.863540 24541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "bd4b6cbe1c564b29b09c7390f066a461"
format_stamp: "Formatted at 2024-05-03 08:18:05 on dist-test-slave-3g67"
server_key: "126e740e3a04e6a07b915441c6792040"
server_key_iv: "2c4259d1645ae3fd6c043172ec0c3772"
server_key_version: "encryptionkey@0"
I20240503 08:18:05.864753 24541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:18:05.865486 24556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:05.865671 24541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:05.865751 24541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "bd4b6cbe1c564b29b09c7390f066a461"
format_stamp: "Formatted at 2024-05-03 08:18:05 on dist-test-slave-3g67"
server_key: "126e740e3a04e6a07b915441c6792040"
server_key_iv: "2c4259d1645ae3fd6c043172ec0c3772"
server_key_version: "encryptionkey@0"
I20240503 08:18:05.865813 24541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:05.878439 24541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:05.880402 24541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:05.880529 24541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:05.884161 24541 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:35113
I20240503 08:18:05.884207 24608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:35113 every 8 connection(s)
I20240503 08:18:05.884507 24541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:05.885205 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 24541
I20240503 08:18:05.885301 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:05.885341 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:05.885494 6419 external_mini_cluster.cc:1428] Setting key 38445e24102ecc8a51bb7e6bec530a6a
I20240503 08:18:05.888363 24609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461: Bootstrap starting.
I20240503 08:18:05.888931 24609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:05.889199 24609 log.cc:824] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:05.890005 24609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461: No bootstrap required, opened a new log
May 03 08:18:05 dist-test-slave-3g67 krb5kdc[24528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:05.891923 24609 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4b6cbe1c564b29b09c7390f066a461" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35113 } }
I20240503 08:18:05.892043 24609 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:05.892060 24609 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4b6cbe1c564b29b09c7390f066a461, State: Initialized, Role: FOLLOWER
I20240503 08:18:05.892153 24609 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4b6cbe1c564b29b09c7390f066a461" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35113 } }
I20240503 08:18:05.892217 24609 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:05.892241 24609 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:05.892288 24609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:05.892987 24609 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4b6cbe1c564b29b09c7390f066a461" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35113 } }
I20240503 08:18:05.893100 24609 leader_election.cc:302] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd4b6cbe1c564b29b09c7390f066a461; no voters:
I20240503 08:18:05.893263 24609 leader_election.cc:288] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:05.893313 24614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:05.893450 24614 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [term 1 LEADER]: Becoming Leader. State: Replica: bd4b6cbe1c564b29b09c7390f066a461, State: Running, Role: LEADER
I20240503 08:18:05.893512 24609 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:05.893627 24614 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4b6cbe1c564b29b09c7390f066a461" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35113 } }
I20240503 08:18:05.893877 24614 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd4b6cbe1c564b29b09c7390f066a461. Latest consensus state: current_term: 1 leader_uuid: "bd4b6cbe1c564b29b09c7390f066a461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4b6cbe1c564b29b09c7390f066a461" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35113 } } }
I20240503 08:18:05.893895 24616 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd4b6cbe1c564b29b09c7390f066a461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4b6cbe1c564b29b09c7390f066a461" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 35113 } } }
I20240503 08:18:05.893996 24614 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:05.894210 24616 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:05.894384 24618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:05.895077 24618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:05.895929 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:05.886015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.6.68.254:35113)
0503 08:18:05.886156 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:05.886165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:05.886183 (+ 18us) server_negotiation.cc:407] Connection header received
0503 08:18:05.886755 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:05.886761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:05.886802 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:05.886916 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:05.887898 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:05.888695 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:05.889461 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:05.889743 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:05.892030 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:05.892037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:05.892038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:05.892060 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:05.894196 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:05.894607 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:05.894610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:05.894611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:05.894670 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:05.894894 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:05.894895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:05.894896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:05.895177 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:05.895345 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:05.895584 (+ 239us) server_negotiation.cc:299] Negotiation successful
0503 08:18:05.895742 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20240503 08:18:05.896814 24618 catalog_manager.cc:1347] Generated new cluster ID: 75046783ddd54099b4530edda3569dc1
I20240503 08:18:05.896912 24618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:05.912971 24618 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:05.913522 24618 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:05.919925 24618 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461: Generated new TSK 0
I20240503 08:18:05.920126 24618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:18:05.951344 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35113
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:06.030516 24635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:06.030696 24635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:06.030735 24635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:06.032070 24635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:06.032114 24635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:06.032153 24635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:18:06.033748 24635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:06.034003 24635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:06.034408 24635 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:06.039580 24640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:06.040071 24643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:06.040638 24641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:06.041208 24635 server_base.cc:1034] running on GCE node
I20240503 08:18:06.041379 24635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:06.041581 24635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:06.042721 24635 hybrid_clock.cc:648] HybridClock initialized: now 1714724286042710 us; error 32 us; skew 500 ppm
May 03 08:18:06 dist-test-slave-3g67 krb5kdc[24528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:06.045315 24635 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:18:06.045797 24635 webserver.cc:460] Webserver started at http://127.6.68.193:35035/ using document root <none> and password file <none>
I20240503 08:18:06.046015 24635 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:06.046062 24635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:06.046160 24635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:06.047389 24635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "4feacb39613342058a52d9161f3b04bc"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "2a2cb230f639ced2a3626c78229f7295"
server_key_iv: "126effe5f85625cfaa6f1f17d6d22392"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.047755 24635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "4feacb39613342058a52d9161f3b04bc"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "2a2cb230f639ced2a3626c78229f7295"
server_key_iv: "126effe5f85625cfaa6f1f17d6d22392"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.049154 24635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:18:06.049928 24650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.050140 24635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:18:06.050217 24635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "4feacb39613342058a52d9161f3b04bc"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "2a2cb230f639ced2a3626c78229f7295"
server_key_iv: "126effe5f85625cfaa6f1f17d6d22392"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.050297 24635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:06.061174 24635 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:06.065182 24635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:06.065289 24635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:06.065502 24635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:06.065779 24635 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:06.065812 24635 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.065834 24635 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:06.065850 24635 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.071278 24635 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:41109
I20240503 08:18:06.071349 24763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:41109 every 8 connection(s)
I20240503 08:18:06.071604 24635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:18:06.076198 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 24635
I20240503 08:18:06.076319 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:18:06.076541 6419 external_mini_cluster.cc:1428] Setting key 0006981adc13e4f88948465208b558bf
May 03 08:18:06 dist-test-slave-3g67 krb5kdc[24528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:06.080355 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.072353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:33051 (local address 127.6.68.254:35113)
0503 08:18:06.072395 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.072397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.072565 (+ 168us) server_negotiation.cc:407] Connection header received
0503 08:18:06.073025 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.073026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.073035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.073098 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:06.073817 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.074298 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.075214 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.075334 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.077442 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:06.077449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:06.077449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:06.077457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:06.079079 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:06.079565 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:06.079567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:06.079567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:06.079593 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:06.079794 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:06.079795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:06.079796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:06.079880 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:06.079937 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.080174 (+ 237us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.080264 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:18:06.080461 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.072293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:35113 (local address 127.6.68.193:33051)
0503 08:18:06.072473 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:06.072501 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:06.072943 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:06.073152 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:06.073156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:06.073445 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:06.073734 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.073738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.074366 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.074367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:06.075132 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.075135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.075235 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.075780 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:06.075795 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:06.077353 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:06.079161 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:06.079168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:06.079170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:06.079481 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:06.079665 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:06.079666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:06.079667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:06.079739 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:06.079934 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:06.079939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:06.080053 (+ 114us) client_negotiation.cc:769] Sending connection context
0503 08:18:06.080147 (+ 94us) client_negotiation.cc:240] Negotiation successful
0503 08:18:06.080275 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20240503 08:18:06.081100 24764 heartbeater.cc:347] Connected to a master server at 127.6.68.254:35113
I20240503 08:18:06.081224 24764 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:06.081472 24764 heartbeater.cc:511] Master 127.6.68.254:35113 requested a full tablet report, sending...
I20240503 08:18:06.082010 24573 ts_manager.cc:194] Registered new tserver with Master: 4feacb39613342058a52d9161f3b04bc (127.6.68.193:41109)
I20240503 08:18:06.082656 24573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:33051
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:18:06.102614 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35113
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:06.188928 24769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:06.189117 24769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:06.189139 24769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:06.190384 24769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:06.190423 24769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:06.190452 24769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:18:06.191983 24769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:06.192153 24769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:06.192301 24769 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:06.196918 24775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:06.197477 24774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:06.198043 24777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:06.198753 24769 server_base.cc:1034] running on GCE node
I20240503 08:18:06.198925 24769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:06.199131 24769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:06.200263 24769 hybrid_clock.cc:648] HybridClock initialized: now 1714724286200241 us; error 35 us; skew 500 ppm
May 03 08:18:06 dist-test-slave-3g67 krb5kdc[24528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:06.202666 24769 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:18:06.203042 24769 webserver.cc:460] Webserver started at http://127.6.68.194:35865/ using document root <none> and password file <none>
I20240503 08:18:06.203222 24769 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:06.203266 24769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:06.203372 24769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:06.204393 24769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "258702b9e99f40968e3fbc35ed3af752"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "2bac1570b5572442f27cf21b4af53662"
server_key_iv: "092fcaf512d3098cfbe7ce3cadaa9b81"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.204700 24769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "258702b9e99f40968e3fbc35ed3af752"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "2bac1570b5572442f27cf21b4af53662"
server_key_iv: "092fcaf512d3098cfbe7ce3cadaa9b81"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.205869 24769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:18:06.206619 24784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.206826 24769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:18:06.206920 24769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "258702b9e99f40968e3fbc35ed3af752"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "2bac1570b5572442f27cf21b4af53662"
server_key_iv: "092fcaf512d3098cfbe7ce3cadaa9b81"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.206977 24769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:06.217762 24769 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:06.219738 24769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:06.219859 24769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:06.220060 24769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:06.220342 24769 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:06.220374 24769 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.220410 24769 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:06.220444 24769 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.225888 24769 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:46197
I20240503 08:18:06.225937 24897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:46197 every 8 connection(s)
I20240503 08:18:06.226220 24769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:18:06.227242 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 24769
I20240503 08:18:06.227353 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:18:06.227535 6419 external_mini_cluster.cc:1428] Setting key 01863f5a9f7d0e68d856d83160df1c48
May 03 08:18:06 dist-test-slave-3g67 krb5kdc[24528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
I20240503 08:18:06.235574 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.226936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:59737 (local address 127.6.68.254:35113)
0503 08:18:06.226994 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.226997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.227186 (+ 189us) server_negotiation.cc:407] Connection header received
0503 08:18:06.227880 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.227881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.227893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.227962 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:06.228869 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.229350 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.230510 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.230635 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.232880 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:06.232887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:06.232887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:06.232895 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:06.234379 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:06.234758 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:06.234760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:06.234760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:06.234784 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:06.234960 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:06.234962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:06.234962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:06.235055 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:06.235115 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.235383 (+ 268us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.235478 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240503 08:18:06.235715 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.226899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:35113 (local address 127.6.68.194:59737)
0503 08:18:06.227047 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:06.227078 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:06.227755 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:06.228021 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:06.228024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:06.228347 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:06.228768 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.228774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.229434 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.229435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:06.230416 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.230418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.230567 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.231115 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:06.231136 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:06.232778 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:06.234481 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:06.234486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:06.234489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:06.234689 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:06.234840 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:06.234841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:06.234841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:06.234904 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:06.235111 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:06.235117 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:06.235244 (+ 127us) client_negotiation.cc:769] Sending connection context
0503 08:18:06.235357 (+ 113us) client_negotiation.cc:240] Negotiation successful
0503 08:18:06.235498 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20240503 08:18:06.236146 24898 heartbeater.cc:347] Connected to a master server at 127.6.68.254:35113
I20240503 08:18:06.236255 24898 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:06.236449 24898 heartbeater.cc:511] Master 127.6.68.254:35113 requested a full tablet report, sending...
I20240503 08:18:06.236845 24573 ts_manager.cc:194] Registered new tserver with Master: 258702b9e99f40968e3fbc35ed3af752 (127.6.68.194:46197)
I20240503 08:18:06.237290 24573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:59737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:18:06.254580 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35113
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:06.334877 24903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:06.335059 24903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:06.335083 24903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:06.336417 24903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:06.336462 24903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:06.336488 24903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:18:06.338043 24903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:35113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:06.338243 24903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:06.338436 24903 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:06.343180 24911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:06.343719 24908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:06.344238 24909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:06.344914 24903 server_base.cc:1034] running on GCE node
I20240503 08:18:06.345108 24903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:06.345333 24903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:06.346513 24903 hybrid_clock.cc:648] HybridClock initialized: now 1714724286346487 us; error 46 us; skew 500 ppm
May 03 08:18:06 dist-test-slave-3g67 krb5kdc[24528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:06.349094 24903 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:18:06.349501 24903 webserver.cc:460] Webserver started at http://127.6.68.195:37097/ using document root <none> and password file <none>
I20240503 08:18:06.349679 24903 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:06.349720 24903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:06.349825 24903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:06.350807 24903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "4689e4c5361e49e5b36a2696e8bb23af"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "ec8dbcf43bdc7b9277700cd1eb122434"
server_key_iv: "54f315b8bc2299a9bac46760666a148c"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.351105 24903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "4689e4c5361e49e5b36a2696e8bb23af"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "ec8dbcf43bdc7b9277700cd1eb122434"
server_key_iv: "54f315b8bc2299a9bac46760666a148c"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.352416 24903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:18:06.353103 24918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.353274 24903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:06.353371 24903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "4689e4c5361e49e5b36a2696e8bb23af"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "ec8dbcf43bdc7b9277700cd1eb122434"
server_key_iv: "54f315b8bc2299a9bac46760666a148c"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.353437 24903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:06.369944 24903 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:06.372020 24903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:06.372145 24903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:06.372341 24903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:06.372620 24903 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:06.372653 24903 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.372690 24903 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:06.372720 24903 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.377882 24903 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:33639
I20240503 08:18:06.377916 25031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:33639 every 8 connection(s)
I20240503 08:18:06.378198 24903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:18:06.379210 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 24903
I20240503 08:18:06.379313 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:18:06.379473 6419 external_mini_cluster.cc:1428] Setting key c6a796de11f651b85d5a26fbc1380e1e
May 03 08:18:06 dist-test-slave-3g67 krb5kdc[24528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:06.386677 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.378885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:48969 (local address 127.6.68.254:35113)
0503 08:18:06.378973 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.378976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.379115 (+ 139us) server_negotiation.cc:407] Connection header received
0503 08:18:06.379708 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.379710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.379722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.379883 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:06.380753 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.381364 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.382300 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.382440 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.384366 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:06.384371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:06.384371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:06.384378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:06.385656 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:06.386011 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:06.386012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:06.386012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:06.386035 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:06.386185 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:06.386185 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:06.386185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:06.386271 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:06.386320 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.386543 (+ 223us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.386610 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20240503 08:18:06.386772 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.378845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:35113 (local address 127.6.68.195:48969)
0503 08:18:06.378990 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:06.379015 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:06.379518 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:06.379949 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:06.379954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:06.380314 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:06.380635 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.380640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.381462 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.381463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:06.382207 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.382209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.382315 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.382840 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:06.382855 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:06.384280 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:06.385737 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:06.385742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:06.385744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:06.385933 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:06.386092 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:06.386093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:06.386093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:06.386139 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:06.386317 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:06.386320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:06.386417 (+ 97us) client_negotiation.cc:769] Sending connection context
0503 08:18:06.386513 (+ 96us) client_negotiation.cc:240] Negotiation successful
0503 08:18:06.386621 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20240503 08:18:06.387284 25032 heartbeater.cc:347] Connected to a master server at 127.6.68.254:35113
I20240503 08:18:06.387367 25032 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:06.387576 25032 heartbeater.cc:511] Master 127.6.68.254:35113 requested a full tablet report, sending...
I20240503 08:18:06.387964 24573 ts_manager.cc:194] Registered new tserver with Master: 4689e4c5361e49e5b36a2696e8bb23af (127.6.68.195:33639)
I20240503 08:18:06.388342 24573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:48969
I20240503 08:18:06.391680 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:18:06.398653 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.393011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.6.68.254:35113)
0503 08:18:06.393072 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.393074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.393211 (+ 137us) server_negotiation.cc:407] Connection header received
0503 08:18:06.393230 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.393231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.393239 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.393287 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:06.394093 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.394621 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.395219 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.395368 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.396526 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:06.396529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:06.396530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:06.396535 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:06.397689 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:06.398021 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:06.398021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:06.398022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:06.398039 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:06.398241 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:06.398242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:06.398242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:06.398329 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:06.398403 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.398511 (+ 108us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.398562 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240503 08:18:06.400470 24573 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:18:06.401937 24573 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:18:06.407205 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.404027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:33639 (local address 127.0.0.1:59410)
0503 08:18:06.404272 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:06.404297 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:06.404374 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:06.404685 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:06.404686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:06.404688 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:06.404931 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.404933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.406053 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.406054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:06.406618 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.406620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.406688 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.406727 (+ 39us) client_negotiation.cc:769] Sending connection context
0503 08:18:06.406782 (+ 55us) client_negotiation.cc:240] Negotiation successful
0503 08:18:06.407126 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":58,"threads_started":1}
I20240503 08:18:06.407701 25042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.404025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:46197 (local address 127.0.0.1:33484)
0503 08:18:06.404272 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:06.404298 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:06.404375 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:06.404596 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:06.404598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:06.404601 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:06.405142 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.405144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.406500 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.406501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:06.407453 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.407455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.407519 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.407523 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:18:06.407559 (+ 36us) client_negotiation.cc:240] Negotiation successful
0503 08:18:06.407605 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":58,"threads_started":1}
I20240503 08:18:06.407724 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.404265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.6.68.195:33639)
0503 08:18:06.404511 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.404525 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.404532 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:06.404567 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.404568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.404585 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.404669 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:06.405099 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.405957 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.406716 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.407226 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.407551 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.407581 (+ 30us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.407653 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":71,"threads_started":1}
I20240503 08:18:06.408288 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.404256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.6.68.194:46197)
0503 08:18:06.404447 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.404454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.404465 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:18:06.404497 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.404498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.404511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.404571 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:06.405230 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.406418 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.407554 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.408106 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.408145 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.408174 (+ 29us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.408211 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20240503 08:18:06.408342 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.404027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:41109 (local address 127.0.0.1:60896)
0503 08:18:06.404652 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:06.404662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:06.404746 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:06.404938 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:06.404939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:06.404941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:06.405138 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.405140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.406489 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.406490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:06.407435 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.407438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.408185 (+ 747us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.408188 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:18:06.408244 (+ 56us) client_negotiation.cc:240] Negotiation successful
0503 08:18:06.408284 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"spinlock_wait_cycles":896,"thread_start_us":64,"threads_started":1}
I20240503 08:18:06.408445 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 5a98e03286b24d20ae1233157942b451 (DEFAULT_TABLE table=test-table [id=e6d1af9f76f6464686adbff423660fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:06.408746 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a98e03286b24d20ae1233157942b451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:06.408839 24832 tablet_service.cc:1467] Processing CreateTablet for tablet 5a98e03286b24d20ae1233157942b451 (DEFAULT_TABLE table=test-table [id=e6d1af9f76f6464686adbff423660fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:06.409117 24832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a98e03286b24d20ae1233157942b451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:06.411309 25046 tablet_bootstrap.cc:492] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: Bootstrap starting.
I20240503 08:18:06.411403 25047 tablet_bootstrap.cc:492] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: Bootstrap starting.
I20240503 08:18:06.411614 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.404105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.6.68.193:41109)
0503 08:18:06.404440 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.404448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.404717 (+ 269us) server_negotiation.cc:407] Connection header received
0503 08:18:06.404831 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.404833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.404849 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.404939 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:06.405229 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.406388 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.407562 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.408115 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.408159 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.408278 (+ 119us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.408322 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":89,"threads_started":1}
I20240503 08:18:06.411986 25046 tablet_bootstrap.cc:654] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:06.412021 25047 tablet_bootstrap.cc:654] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:06.412220 24698 tablet_service.cc:1467] Processing CreateTablet for tablet 5a98e03286b24d20ae1233157942b451 (DEFAULT_TABLE table=test-table [id=e6d1af9f76f6464686adbff423660fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:06.412309 25047 log.cc:824] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:06.412408 25046 log.cc:824] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:06.412503 24698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a98e03286b24d20ae1233157942b451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:06.413064 25047 tablet_bootstrap.cc:492] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: No bootstrap required, opened a new log
I20240503 08:18:06.413131 25047 ts_tablet_manager.cc:1367] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:18:06.415519 25047 raft_consensus.cc:348] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feacb39613342058a52d9161f3b04bc" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 41109 } } peers { permanent_uuid: "258702b9e99f40968e3fbc35ed3af752" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 46197 } } peers { permanent_uuid: "4689e4c5361e49e5b36a2696e8bb23af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 33639 } }
I20240503 08:18:06.415645 25050 tablet_bootstrap.cc:492] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc: Bootstrap starting.
I20240503 08:18:06.415709 25047 raft_consensus.cc:374] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:06.415741 25047 raft_consensus.cc:729] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 258702b9e99f40968e3fbc35ed3af752, State: Initialized, Role: FOLLOWER
I20240503 08:18:06.415839 25047 consensus_queue.cc:260] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feacb39613342058a52d9161f3b04bc" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 41109 } } peers { permanent_uuid: "258702b9e99f40968e3fbc35ed3af752" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 46197 } } peers { permanent_uuid: "4689e4c5361e49e5b36a2696e8bb23af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 33639 } }
I20240503 08:18:06.416208 25050 tablet_bootstrap.cc:654] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:06.416208 25047 ts_tablet_manager.cc:1398] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:18:06.416286 24898 heartbeater.cc:503] Master 127.6.68.254:35113 was elected leader, sending a full tablet report...
I20240503 08:18:06.416497 25050 log.cc:824] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:06.416795 25046 tablet_bootstrap.cc:492] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: No bootstrap required, opened a new log
I20240503 08:18:06.416841 25046 ts_tablet_manager.cc:1367] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 08:18:06.417212 25050 tablet_bootstrap.cc:492] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc: No bootstrap required, opened a new log
I20240503 08:18:06.417263 25050 ts_tablet_manager.cc:1367] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:18:06.419292 25046 raft_consensus.cc:348] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feacb39613342058a52d9161f3b04bc" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 41109 } } peers { permanent_uuid: "258702b9e99f40968e3fbc35ed3af752" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 46197 } } peers { permanent_uuid: "4689e4c5361e49e5b36a2696e8bb23af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 33639 } }
I20240503 08:18:06.419430 25046 raft_consensus.cc:374] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:06.419464 25046 raft_consensus.cc:729] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4689e4c5361e49e5b36a2696e8bb23af, State: Initialized, Role: FOLLOWER
I20240503 08:18:06.419529 25050 raft_consensus.cc:348] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feacb39613342058a52d9161f3b04bc" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 41109 } } peers { permanent_uuid: "258702b9e99f40968e3fbc35ed3af752" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 46197 } } peers { permanent_uuid: "4689e4c5361e49e5b36a2696e8bb23af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 33639 } }
I20240503 08:18:06.419567 25046 consensus_queue.cc:260] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feacb39613342058a52d9161f3b04bc" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 41109 } } peers { permanent_uuid: "258702b9e99f40968e3fbc35ed3af752" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 46197 } } peers { permanent_uuid: "4689e4c5361e49e5b36a2696e8bb23af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 33639 } }
I20240503 08:18:06.419656 25050 raft_consensus.cc:374] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:06.419682 25050 raft_consensus.cc:729] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4feacb39613342058a52d9161f3b04bc, State: Initialized, Role: FOLLOWER
I20240503 08:18:06.419783 25050 consensus_queue.cc:260] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feacb39613342058a52d9161f3b04bc" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 41109 } } peers { permanent_uuid: "258702b9e99f40968e3fbc35ed3af752" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 46197 } } peers { permanent_uuid: "4689e4c5361e49e5b36a2696e8bb23af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 33639 } }
I20240503 08:18:06.419982 25046 ts_tablet_manager.cc:1398] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:18:06.420024 25050 ts_tablet_manager.cc:1398] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:18:06.420114 24764 heartbeater.cc:503] Master 127.6.68.254:35113 was elected leader, sending a full tablet report...
I20240503 08:18:06.420111 25032 heartbeater.cc:503] Master 127.6.68.254:35113 was elected leader, sending a full tablet report...
W20240503 08:18:06.476724 24899 tablet.cc:2369] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:18:06.572321 24765 tablet.cc:2369] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:18:06.611196 25051 raft_consensus.cc:482] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:18:06.611321 25051 raft_consensus.cc:504] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feacb39613342058a52d9161f3b04bc" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 41109 } } peers { permanent_uuid: "258702b9e99f40968e3fbc35ed3af752" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 46197 } } peers { permanent_uuid: "4689e4c5361e49e5b36a2696e8bb23af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 33639 } }
I20240503 08:18:06.611649 25051 leader_election.cc:288] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4feacb39613342058a52d9161f3b04bc (127.6.68.193:41109), 4689e4c5361e49e5b36a2696e8bb23af (127.6.68.195:33639)
I20240503 08:18:06.613699 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.611819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:41109 (local address 127.6.68.194:43883)
0503 08:18:06.611870 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:06.611883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:06.611934 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:06.612098 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:06.612099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:06.612100 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:06.612282 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.612284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.612938 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.612939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:06.613488 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.613489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.613559 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.613563 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:18:06.613596 (+ 33us) client_negotiation.cc:240] Negotiation successful
0503 08:18:06.613628 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20240503 08:18:06.614084 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.611903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:43883 (local address 127.6.68.193:41109)
0503 08:18:06.611983 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.611986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.611995 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:06.612024 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.612025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.612036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.612088 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:06.612371 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.612863 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.613574 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.613991 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.613998 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.614017 (+ 19us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.614037 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240503 08:18:06.614373 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a98e03286b24d20ae1233157942b451" candidate_uuid: "258702b9e99f40968e3fbc35ed3af752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4feacb39613342058a52d9161f3b04bc" is_pre_election: true
I20240503 08:18:06.614480 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.611826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:33639 (local address 127.6.68.194:44969)
0503 08:18:06.612045 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:06.612054 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:06.612128 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:06.612264 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:06.612264 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:06.612266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:06.612508 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.612510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.613144 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.613145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:06.613751 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.613755 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.614303 (+ 548us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.614307 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:18:06.614363 (+ 56us) client_negotiation.cc:240] Negotiation successful
0503 08:18:06.614400 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":10,"thread_start_us":98,"threads_started":1}
I20240503 08:18:06.614475 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.611956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:44969 (local address 127.6.68.195:33639)
0503 08:18:06.612024 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.612026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.612121 (+ 95us) server_negotiation.cc:407] Connection header received
0503 08:18:06.612196 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.612196 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.612205 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.612257 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:06.612576 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.613053 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.613863 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.614268 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.614275 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.614380 (+ 105us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.614416 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:18:06.614521 24718 raft_consensus.cc:2454] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 258702b9e99f40968e3fbc35ed3af752 in term 0.
I20240503 08:18:06.614725 24785 leader_election.cc:302] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 258702b9e99f40968e3fbc35ed3af752, 4feacb39613342058a52d9161f3b04bc; no voters:
I20240503 08:18:06.614837 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a98e03286b24d20ae1233157942b451" candidate_uuid: "258702b9e99f40968e3fbc35ed3af752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4689e4c5361e49e5b36a2696e8bb23af" is_pre_election: true
I20240503 08:18:06.614863 25051 raft_consensus.cc:2781] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:18:06.614933 25051 raft_consensus.cc:482] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:18:06.614964 25051 raft_consensus.cc:3037] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:06.614966 24986 raft_consensus.cc:2454] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 258702b9e99f40968e3fbc35ed3af752 in term 0.
I20240503 08:18:06.615690 25051 raft_consensus.cc:504] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feacb39613342058a52d9161f3b04bc" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 41109 } } peers { permanent_uuid: "258702b9e99f40968e3fbc35ed3af752" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 46197 } } peers { permanent_uuid: "4689e4c5361e49e5b36a2696e8bb23af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 33639 } }
I20240503 08:18:06.615832 25051 leader_election.cc:288] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [CANDIDATE]: Term 1 election: Requested vote from peers 4feacb39613342058a52d9161f3b04bc (127.6.68.193:41109), 4689e4c5361e49e5b36a2696e8bb23af (127.6.68.195:33639)
I20240503 08:18:06.615975 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a98e03286b24d20ae1233157942b451" candidate_uuid: "258702b9e99f40968e3fbc35ed3af752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4feacb39613342058a52d9161f3b04bc"
I20240503 08:18:06.616021 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a98e03286b24d20ae1233157942b451" candidate_uuid: "258702b9e99f40968e3fbc35ed3af752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4689e4c5361e49e5b36a2696e8bb23af"
I20240503 08:18:06.616065 24718 raft_consensus.cc:3037] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:06.616077 24986 raft_consensus.cc:3037] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:06.616938 24986 raft_consensus.cc:2454] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 258702b9e99f40968e3fbc35ed3af752 in term 1.
I20240503 08:18:06.616956 24718 raft_consensus.cc:2454] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 258702b9e99f40968e3fbc35ed3af752 in term 1.
I20240503 08:18:06.617137 24788 leader_election.cc:302] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 258702b9e99f40968e3fbc35ed3af752, 4689e4c5361e49e5b36a2696e8bb23af; no voters:
I20240503 08:18:06.617246 25051 raft_consensus.cc:2781] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:06.617421 25051 raft_consensus.cc:686] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 1 LEADER]: Becoming Leader. State: Replica: 258702b9e99f40968e3fbc35ed3af752, State: Running, Role: LEADER
I20240503 08:18:06.617496 25051 consensus_queue.cc:237] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feacb39613342058a52d9161f3b04bc" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 41109 } } peers { permanent_uuid: "258702b9e99f40968e3fbc35ed3af752" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 46197 } } peers { permanent_uuid: "4689e4c5361e49e5b36a2696e8bb23af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 33639 } }
I20240503 08:18:06.618300 24573 catalog_manager.cc:5484] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 reported cstate change: term changed from 0 to 1, leader changed from <none> to 258702b9e99f40968e3fbc35ed3af752 (127.6.68.194). New cstate: current_term: 1 leader_uuid: "258702b9e99f40968e3fbc35ed3af752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feacb39613342058a52d9161f3b04bc" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 41109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "258702b9e99f40968e3fbc35ed3af752" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 46197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4689e4c5361e49e5b36a2696e8bb23af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 33639 } health_report { overall_health: UNKNOWN } } }
W20240503 08:18:06.628720 25033 tablet.cc:2369] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:18:06.634889 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.632833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.6.68.194:46197)
0503 08:18:06.632891 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.632893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.632900 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:06.632938 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.632939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.632955 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.633017 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:06.633241 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.633725 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.634233 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.634357 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.634511 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:06.634587 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:06.634650 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.634793 (+ 143us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.634800 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240503 08:18:06.636663 24986 raft_consensus.cc:1261] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af [term 1 FOLLOWER]: Refusing update from remote peer 258702b9e99f40968e3fbc35ed3af752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:06.636696 24718 raft_consensus.cc:1261] T 5a98e03286b24d20ae1233157942b451 P 4feacb39613342058a52d9161f3b04bc [term 1 FOLLOWER]: Refusing update from remote peer 258702b9e99f40968e3fbc35ed3af752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:06.637005 25056 consensus_queue.cc:1035] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4689e4c5361e49e5b36a2696e8bb23af" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:06.637141 25051 consensus_queue.cc:1035] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4feacb39613342058a52d9161f3b04bc" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 41109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:06.639925 25060 mvcc.cc:205] Tried to move back new op lower bound from 7023510678061150208 to 7023510677985861632. Current Snapshot: MvccSnapshot[applied={T|T < 7023510678061150208}]
I20240503 08:18:06.640444 25059 mvcc.cc:205] Tried to move back new op lower bound from 7023510678061150208 to 7023510677985861632. Current Snapshot: MvccSnapshot[applied={T|T < 7023510678061150208 or (T in {7023510678061150208})}]
I20240503 08:18:06.642354 24573 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57094:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:06.642448 24573 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57094:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:06.643139 24573 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd4b6cbe1c564b29b09c7390f066a461: Sending DeleteTablet for 3 replicas of tablet 5a98e03286b24d20ae1233157942b451
I20240503 08:18:06.643496 24698 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a98e03286b24d20ae1233157942b451 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:06 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:60896
I20240503 08:18:06.643520 24832 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a98e03286b24d20ae1233157942b451 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:06 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:33484
I20240503 08:18:06.643774 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 24635
I20240503 08:18:06.644285 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a98e03286b24d20ae1233157942b451 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:06 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:59410
I20240503 08:18:06.645059 25072 tablet_replica.cc:330] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: stopping tablet replica
I20240503 08:18:06.645184 25072 raft_consensus.cc:2229] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:18:06.645320 25072 raft_consensus.cc:2258] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:06.645776 25072 ts_tablet_manager.cc:1875] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:06.646023 25071 tablet_replica.cc:330] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: stopping tablet replica
I20240503 08:18:06.646104 25071 raft_consensus.cc:2229] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:18:06.646211 25071 raft_consensus.cc:2258] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:06.646602 25071 ts_tablet_manager.cc:1875] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:06.647692 25072 ts_tablet_manager.cc:1888] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:18:06.647764 25072 log.cc:1196] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/5a98e03286b24d20ae1233157942b451
I20240503 08:18:06.648059 25072 ts_tablet_manager.cc:1909] T 5a98e03286b24d20ae1233157942b451 P 4689e4c5361e49e5b36a2696e8bb23af: Deleting consensus metadata
I20240503 08:18:06.648449 24560 catalog_manager.cc:4830] TS 4689e4c5361e49e5b36a2696e8bb23af (127.6.68.195:33639): tablet 5a98e03286b24d20ae1233157942b451 (table test-table [id=e6d1af9f76f6464686adbff423660fa8]) successfully deleted
W20240503 08:18:06.651039 24557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:41109 (error 108)
I20240503 08:18:06.651141 25071 ts_tablet_manager.cc:1888] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:18:06.651216 25071 log.cc:1196] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/5a98e03286b24d20ae1233157942b451
I20240503 08:18:06.651508 25071 ts_tablet_manager.cc:1909] T 5a98e03286b24d20ae1233157942b451 P 258702b9e99f40968e3fbc35ed3af752: Deleting consensus metadata
I20240503 08:18:06.651875 24558 catalog_manager.cc:4830] TS 258702b9e99f40968e3fbc35ed3af752 (127.6.68.194:46197): tablet 5a98e03286b24d20ae1233157942b451 (table test-table [id=e6d1af9f76f6464686adbff423660fa8]) successfully deleted
W20240503 08:18:06.651998 25040 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:06.651511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:41109 (local address 127.0.0.1:60908)
0503 08:18:06.651578 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:06.651590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:06.651707 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:06.651882 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:41109: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":44}
W20240503 08:18:06.652168 24557 catalog_manager.cc:4573] TS 4feacb39613342058a52d9161f3b04bc (127.6.68.193:41109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a98e03286b24d20ae1233157942b451: Network error: Client connection negotiation failed: client connection to 127.6.68.193:41109: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:18:06.652333 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 24769
I20240503 08:18:06.658341 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 24903
I20240503 08:18:06.662927 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 24541
2024-05-03T08:18:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1971 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2024-05-03T08:18:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:06Z Disabled control of system clock
I20240503 08:18:06.690652 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34503
--webserver_interface=127.6.68.254
--webserver_port=0
--builtin_ntp_servers=127.6.68.212:43277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:34503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240503 08:18:06.767599 25077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:06.767804 25077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:06.767824 25077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:06.769065 25077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:06.769100 25077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:06.769112 25077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:06.769124 25077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:06.769134 25077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:06.770501 25077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:34503
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:06.770660 25077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:06.770853 25077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:06.775332 25082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:06.775929 25085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:06.776434 25083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:06.777122 25077 server_base.cc:1034] running on GCE node
I20240503 08:18:06.777343 25077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:06.777649 25077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:06.778786 25077 hybrid_clock.cc:648] HybridClock initialized: now 1714724286778768 us; error 37 us; skew 500 ppm
I20240503 08:18:06.780004 25077 webserver.cc:460] Webserver started at http://127.6.68.254:36305/ using document root <none> and password file <none>
I20240503 08:18:06.780203 25077 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:06.780251 25077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:06.780359 25077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:06.781385 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "1b5f930cd65149b4add2f30907142c31"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "9db8abbdcd1ff27f779877b64a130c3c"
server_key_iv: "c74812b60885829c1fe870e7e7b54793"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.781734 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "1b5f930cd65149b4add2f30907142c31"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "9db8abbdcd1ff27f779877b64a130c3c"
server_key_iv: "c74812b60885829c1fe870e7e7b54793"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.782896 25077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:18:06.783730 25091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.783913 25077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:06.783998 25077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "1b5f930cd65149b4add2f30907142c31"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "9db8abbdcd1ff27f779877b64a130c3c"
server_key_iv: "c74812b60885829c1fe870e7e7b54793"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.784080 25077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:06.791412 25077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:06.791725 25077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:06.791841 25077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:06.795329 25077 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:34503
I20240503 08:18:06.795380 25143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:34503 every 8 connection(s)
I20240503 08:18:06.795707 25077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:06.796303 25144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:06.798694 25144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31: Bootstrap starting.
I20240503 08:18:06.799225 25144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:06.799466 25144 log.cc:824] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:06.800143 25144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31: No bootstrap required, opened a new log
I20240503 08:18:06.801517 25144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5f930cd65149b4add2f30907142c31" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34503 } }
I20240503 08:18:06.801635 25144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:06.801652 25144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5f930cd65149b4add2f30907142c31, State: Initialized, Role: FOLLOWER
I20240503 08:18:06.801761 25144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5f930cd65149b4add2f30907142c31" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34503 } }
I20240503 08:18:06.801827 25144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:06.801849 25144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:06.801892 25144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:06.802423 25144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5f930cd65149b4add2f30907142c31" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34503 } }
I20240503 08:18:06.802508 25144 leader_election.cc:302] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b5f930cd65149b4add2f30907142c31; no voters:
I20240503 08:18:06.802649 25144 leader_election.cc:288] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:06.802752 25147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:06.802887 25147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [term 1 LEADER]: Becoming Leader. State: Replica: 1b5f930cd65149b4add2f30907142c31, State: Running, Role: LEADER
I20240503 08:18:06.802927 25144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:06.803035 25147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5f930cd65149b4add2f30907142c31" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34503 } }
I20240503 08:18:06.803340 25149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b5f930cd65149b4add2f30907142c31. Latest consensus state: current_term: 1 leader_uuid: "1b5f930cd65149b4add2f30907142c31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5f930cd65149b4add2f30907142c31" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34503 } } }
I20240503 08:18:06.803437 25149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:06.803335 25148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b5f930cd65149b4add2f30907142c31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5f930cd65149b4add2f30907142c31" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34503 } } }
I20240503 08:18:06.803699 25152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:06.803714 25148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:06.804222 25152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:06.804513 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 25077
I20240503 08:18:06.804625 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:06.804808 6419 external_mini_cluster.cc:1428] Setting key b7928197e735d8555db25d9c60392616
I20240503 08:18:06.805675 25152 catalog_manager.cc:1347] Generated new cluster ID: aaf76d9de76e47fa9546850465cb6eb7
I20240503 08:18:06.805727 25152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:06.809769 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.805418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.6.68.254:34503)
0503 08:18:06.805605 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.805613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.805837 (+ 224us) server_negotiation.cc:407] Connection header received
0503 08:18:06.805969 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.805975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.806038 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.806173 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:06.806604 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.807746 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.808541 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.808768 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.809191 (+ 423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:06.809196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:06.809197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:06.809384 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:06.809409 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:06.809416 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:06.809488 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.809566 (+ 78us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.809612 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20240503 08:18:06.818993 25152 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:06.819747 25152 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:06.829098 25152 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31: Generated new TSK 0
I20240503 08:18:06.829293 25152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:06.836707 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34503
--builtin_ntp_servers=127.6.68.212:43277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240503 08:18:06.917069 25168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:06.917275 25168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:06.917312 25168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:06.918656 25168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:06.918705 25168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:06.918746 25168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:18:06.920223 25168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:06.920475 25168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:06.920745 25168 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:06.925946 25174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:06.926329 25173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:06.927021 25176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:06.927654 25168 server_base.cc:1034] running on GCE node
I20240503 08:18:06.927857 25168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:06.928122 25168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:06.929296 25168 hybrid_clock.cc:648] HybridClock initialized: now 1714724286929275 us; error 38 us; skew 500 ppm
I20240503 08:18:06.930891 25168 webserver.cc:460] Webserver started at http://127.6.68.193:45725/ using document root <none> and password file <none>
I20240503 08:18:06.931120 25168 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:06.931185 25168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:06.931317 25168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:06.932693 25168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "34ed4ae82cca47a7a409e0380575170e"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "6cb89464925b85929c63922afd51b941"
server_key_iv: "4945c21b68756f30cf0a15366b40c161"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.933043 25168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "34ed4ae82cca47a7a409e0380575170e"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "6cb89464925b85929c63922afd51b941"
server_key_iv: "4945c21b68756f30cf0a15366b40c161"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.934478 25168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:18:06.935281 25182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.935453 25168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:06.935542 25168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "34ed4ae82cca47a7a409e0380575170e"
format_stamp: "Formatted at 2024-05-03 08:18:06 on dist-test-slave-3g67"
server_key: "6cb89464925b85929c63922afd51b941"
server_key_iv: "4945c21b68756f30cf0a15366b40c161"
server_key_version: "encryptionkey@0"
I20240503 08:18:06.935612 25168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:06.974857 25168 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:06.975129 25168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:06.975217 25168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:06.975402 25168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:06.975736 25168 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:06.975772 25168 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.975796 25168 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:06.975823 25168 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:06.981355 25168 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:38291
I20240503 08:18:06.981415 25295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:38291 every 8 connection(s)
I20240503 08:18:06.981701 25168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:18:06.985992 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.982394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:42117 (local address 127.6.68.254:34503)
0503 08:18:06.982458 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:06.982460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:06.982648 (+ 188us) server_negotiation.cc:407] Connection header received
0503 08:18:06.983072 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:06.983073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:06.983084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:06.983147 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:06.983653 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.984147 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.984677 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.984775 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.985763 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:06.985767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:06.985767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:06.985771 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:06.985785 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:06.985788 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:06.985831 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:06.985901 (+ 70us) server_negotiation.cc:299] Negotiation successful
0503 08:18:06.985932 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240503 08:18:06.986078 25298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:06.982354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34503 (local address 127.6.68.193:42117)
0503 08:18:06.982552 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:06.982575 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:06.982993 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:06.983229 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:06.983232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:06.983240 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:06.983549 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.983552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.984232 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:06.984233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:06.984612 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:06.984613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:06.984720 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:06.985643 (+ 923us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:18:06.985644 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:06.985661 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:18:06.985678 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:06.985838 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:06.985840 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:06.985842 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:18:06.985892 (+ 50us) client_negotiation.cc:240] Negotiation successful
0503 08:18:06.985949 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20240503 08:18:06.986539 25296 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34503
I20240503 08:18:06.986622 25296 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:06.986779 25296 heartbeater.cc:511] Master 127.6.68.254:34503 requested a full tablet report, sending...
I20240503 08:18:06.987236 25108 ts_manager.cc:194] Registered new tserver with Master: 34ed4ae82cca47a7a409e0380575170e (127.6.68.193:38291)
I20240503 08:18:06.987870 25108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.193:42117
I20240503 08:18:06.991808 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 25168
I20240503 08:18:06.991891 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:18:06.992017 6419 external_mini_cluster.cc:1428] Setting key 4692be4eb871afb8b649b800d77b936b
I20240503 08:18:06.992969 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34503
--builtin_ntp_servers=127.6.68.212:43277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240503 08:18:07.070475 25299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:07.070668 25299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:07.070689 25299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:07.071971 25299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:07.072007 25299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:07.072031 25299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:18:07.073405 25299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:07.073590 25299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:07.073792 25299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:07.078850 25305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:07.079077 25307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:07.079782 25304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:07.080283 25299 server_base.cc:1034] running on GCE node
I20240503 08:18:07.080456 25299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:07.080638 25299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:07.081764 25299 hybrid_clock.cc:648] HybridClock initialized: now 1714724287081746 us; error 35 us; skew 500 ppm
I20240503 08:18:07.088742 25299 webserver.cc:460] Webserver started at http://127.6.68.194:36671/ using document root <none> and password file <none>
I20240503 08:18:07.088966 25299 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:07.089015 25299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:07.089123 25299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:07.090135 25299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13"
format_stamp: "Formatted at 2024-05-03 08:18:07 on dist-test-slave-3g67"
server_key: "b3a99416950a0e5d4a0fbb90f01b0997"
server_key_iv: "48ddd79dc254d442e4ef5c137fcf239a"
server_key_version: "encryptionkey@0"
I20240503 08:18:07.090444 25299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13"
format_stamp: "Formatted at 2024-05-03 08:18:07 on dist-test-slave-3g67"
server_key: "b3a99416950a0e5d4a0fbb90f01b0997"
server_key_iv: "48ddd79dc254d442e4ef5c137fcf239a"
server_key_version: "encryptionkey@0"
I20240503 08:18:07.091845 25299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240503 08:18:07.092599 25313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:07.092758 25299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:07.092844 25299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13"
format_stamp: "Formatted at 2024-05-03 08:18:07 on dist-test-slave-3g67"
server_key: "b3a99416950a0e5d4a0fbb90f01b0997"
server_key_iv: "48ddd79dc254d442e4ef5c137fcf239a"
server_key_version: "encryptionkey@0"
I20240503 08:18:07.092921 25299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:07.106189 25299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:07.106457 25299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:07.106573 25299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:07.106775 25299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:07.107069 25299 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:07.107101 25299 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:07.107137 25299 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:07.107182 25299 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:07.112354 25299 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:43953
I20240503 08:18:07.112406 25426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:43953 every 8 connection(s)
I20240503 08:18:07.112727 25299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:18:07.117202 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.113445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:54933 (local address 127.6.68.254:34503)
0503 08:18:07.113538 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:07.113540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:07.113698 (+ 158us) server_negotiation.cc:407] Connection header received
0503 08:18:07.114197 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:07.114198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:07.114211 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:07.114274 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:07.114739 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.115225 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.115780 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.115950 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.116918 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:07.116922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:07.116923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:07.116930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:07.116946 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:07.116949 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:07.116997 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:07.117081 (+ 84us) server_negotiation.cc:299] Negotiation successful
0503 08:18:07.117120 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20240503 08:18:07.117362 25429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.113371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34503 (local address 127.6.68.194:54933)
0503 08:18:07.113598 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:07.113622 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:07.114092 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:07.114338 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:07.114342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:07.114350 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:07.114656 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.114659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.115303 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.115306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:07.115699 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.115701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.115826 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.116791 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:18:07.116792 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:07.116811 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:18:07.116831 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:07.117004 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:07.117008 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:07.117011 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:18:07.117084 (+ 73us) client_negotiation.cc:240] Negotiation successful
0503 08:18:07.117166 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20240503 08:18:07.117470 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 25299
I20240503 08:18:07.117596 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:18:07.117789 6419 external_mini_cluster.cc:1428] Setting key 9983be3cbf202477602591bada3123bd
I20240503 08:18:07.117944 25427 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34503
I20240503 08:18:07.118055 25427 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:07.118273 25427 heartbeater.cc:511] Master 127.6.68.254:34503 requested a full tablet report, sending...
I20240503 08:18:07.118659 25108 ts_manager.cc:194] Registered new tserver with Master: 8cdcbe62c1ee41fb8e016f2ffb8c6b13 (127.6.68.194:43953)
I20240503 08:18:07.118865 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34503
--builtin_ntp_servers=127.6.68.212:43277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240503 08:18:07.119181 25108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.194:54933
W20240503 08:18:07.198428 25430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:07.198627 25430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:07.198661 25430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:07.199970 25430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:07.200011 25430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:07.200047 25430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:18:07.201499 25430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:43277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:07.201716 25430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:07.201948 25430 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:07.206861 25438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:07.207296 25436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:07.207962 25435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:07.208563 25430 server_base.cc:1034] running on GCE node
I20240503 08:18:07.208731 25430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:07.208938 25430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:07.210096 25430 hybrid_clock.cc:648] HybridClock initialized: now 1714724287210076 us; error 36 us; skew 500 ppm
I20240503 08:18:07.211421 25430 webserver.cc:460] Webserver started at http://127.6.68.195:41697/ using document root <none> and password file <none>
I20240503 08:18:07.211660 25430 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:07.211719 25430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:07.211827 25430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:07.212842 25430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "1b0684b1ed7146e882af6d1059291d24"
format_stamp: "Formatted at 2024-05-03 08:18:07 on dist-test-slave-3g67"
server_key: "36672bfb655256e0dce0ff264829d4f5"
server_key_iv: "1d83a875b738293a9bdc4f85d605a41c"
server_key_version: "encryptionkey@0"
I20240503 08:18:07.213153 25430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "1b0684b1ed7146e882af6d1059291d24"
format_stamp: "Formatted at 2024-05-03 08:18:07 on dist-test-slave-3g67"
server_key: "36672bfb655256e0dce0ff264829d4f5"
server_key_iv: "1d83a875b738293a9bdc4f85d605a41c"
server_key_version: "encryptionkey@0"
I20240503 08:18:07.214313 25430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:07.214999 25444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:07.215189 25430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:07.215270 25430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "1b0684b1ed7146e882af6d1059291d24"
format_stamp: "Formatted at 2024-05-03 08:18:07 on dist-test-slave-3g67"
server_key: "36672bfb655256e0dce0ff264829d4f5"
server_key_iv: "1d83a875b738293a9bdc4f85d605a41c"
server_key_version: "encryptionkey@0"
I20240503 08:18:07.215334 25430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:07.232770 25430 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:07.233071 25430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:07.233171 25430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:07.233389 25430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:07.233737 25430 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:07.233773 25430 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:07.233814 25430 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:07.233844 25430 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:07.239218 25430 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:43241
I20240503 08:18:07.239266 25557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:43241 every 8 connection(s)
I20240503 08:18:07.239591 25430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:18:07.243649 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 25430
I20240503 08:18:07.243749 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:18:07.243909 6419 external_mini_cluster.cc:1428] Setting key 1c4d01d14f787ccaf6cad50c6203fedf
I20240503 08:18:07.244396 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.240341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:48869 (local address 127.6.68.254:34503)
0503 08:18:07.240419 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:07.240422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:07.240700 (+ 278us) server_negotiation.cc:407] Connection header received
0503 08:18:07.241219 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:07.241219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:07.241230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:07.241286 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:07.241845 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.242336 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.242895 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.243037 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.244082 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:07.244087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:07.244087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:07.244091 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:07.244107 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:07.244111 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:07.244172 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:07.244273 (+ 101us) server_negotiation.cc:299] Negotiation successful
0503 08:18:07.244311 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240503 08:18:07.244619 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.240317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34503 (local address 127.6.68.195:48869)
0503 08:18:07.240575 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:07.240600 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:07.241079 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:07.241337 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:07.241341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:07.241349 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:07.241688 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.241691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.242436 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.242436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:07.242806 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.242807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.242915 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.243955 (+ 1040us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:18:07.243956 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:07.243984 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:18:07.244006 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:07.244180 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:07.244183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:07.244186 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:18:07.244275 (+ 89us) client_negotiation.cc:240] Negotiation successful
0503 08:18:07.244375 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20240503 08:18:07.245036 25558 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34503
I20240503 08:18:07.245132 25558 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:07.245334 25558 heartbeater.cc:511] Master 127.6.68.254:34503 requested a full tablet report, sending...
I20240503 08:18:07.245743 25108 ts_manager.cc:194] Registered new tserver with Master: 1b0684b1ed7146e882af6d1059291d24 (127.6.68.195:43241)
I20240503 08:18:07.246120 25108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.195:48869
I20240503 08:18:07.256114 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:18:07.260232 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.257637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.6.68.254:34503)
0503 08:18:07.257694 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:07.257696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:07.257786 (+ 90us) server_negotiation.cc:407] Connection header received
0503 08:18:07.257838 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:07.257838 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:07.257847 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:07.257896 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:07.258199 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.258675 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.259240 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.259346 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.259992 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:07.259995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:07.259996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:07.260000 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:07.260013 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:07.260017 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:07.260081 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:07.260157 (+ 76us) server_negotiation.cc:299] Negotiation successful
0503 08:18:07.260169 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240503 08:18:07.262190 25108 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:18:07.263667 25108 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:18:07.269187 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.265498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:43241 (local address 127.0.0.1:47038)
0503 08:18:07.265934 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:07.265945 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:07.266019 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:07.266295 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:07.266300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:07.266321 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:07.266618 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.266620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.268093 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.268095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:07.268958 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.268960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.269019 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.269022 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:18:07.269053 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:18:07.269092 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20240503 08:18:07.269085 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.265498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:38291 (local address 127.0.0.1:42116)
0503 08:18:07.265794 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:07.265820 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:07.265893 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:07.266664 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:07.266665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:07.266667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:07.266865 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.266868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.268212 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.268213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:07.268842 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.268844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.268920 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.268924 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:18:07.268960 (+ 36us) client_negotiation.cc:240] Negotiation successful
0503 08:18:07.268999 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":77,"threads_started":1}
I20240503 08:18:07.269716 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.265736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.6.68.194:43953)
0503 08:18:07.266157 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:07.266170 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:07.266179 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:07.266230 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:07.266232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:07.266250 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:07.266362 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:07.266723 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.267577 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.268957 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.269491 (+ 534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.269568 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:07.269601 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:18:07.269635 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":107,"threads_started":1}
I20240503 08:18:07.269742 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.265808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.6.68.193:38291)
0503 08:18:07.266512 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:07.266517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:07.266525 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:18:07.266557 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:07.266558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:07.266571 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:07.266647 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:07.266980 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.268129 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.268931 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.269462 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.269555 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:07.269590 (+ 35us) server_negotiation.cc:299] Negotiation successful
0503 08:18:07.269647 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":100,"threads_started":1}
I20240503 08:18:07.269852 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.265694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.6.68.195:43241)
0503 08:18:07.265873 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:07.265881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:07.266028 (+ 147us) server_negotiation.cc:407] Connection header received
0503 08:18:07.266110 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:07.266112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:07.266136 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:07.266231 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:07.266759 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.268004 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.269127 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.269632 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.269694 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:07.269723 (+ 29us) server_negotiation.cc:299] Negotiation successful
0503 08:18:07.269764 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20240503 08:18:07.270035 25568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.265519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:43953 (local address 127.0.0.1:38024)
0503 08:18:07.265952 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:07.265961 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:07.266004 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:07.266396 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:07.266397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:07.266399 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:07.266618 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.266620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.267708 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.267711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:07.268450 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.268452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.268521 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.268561 (+ 40us) client_negotiation.cc:769] Sending connection context
0503 08:18:07.268636 (+ 75us) client_negotiation.cc:240] Negotiation successful
0503 08:18:07.268676 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":103,"threads_started":1}
I20240503 08:18:07.270438 25230 tablet_service.cc:1467] Processing CreateTablet for tablet 3587153e23684f8f9ef5a0f8942cca62 (DEFAULT_TABLE table=test-table [id=33360a4d40564ba18f9d86b1411d6584]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:07.270442 25492 tablet_service.cc:1467] Processing CreateTablet for tablet 3587153e23684f8f9ef5a0f8942cca62 (DEFAULT_TABLE table=test-table [id=33360a4d40564ba18f9d86b1411d6584]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:07.270551 25361 tablet_service.cc:1467] Processing CreateTablet for tablet 3587153e23684f8f9ef5a0f8942cca62 (DEFAULT_TABLE table=test-table [id=33360a4d40564ba18f9d86b1411d6584]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:07.270732 25230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3587153e23684f8f9ef5a0f8942cca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:07.270735 25492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3587153e23684f8f9ef5a0f8942cca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:07.270767 25361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3587153e23684f8f9ef5a0f8942cca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:07.273196 25572 tablet_bootstrap.cc:492] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e: Bootstrap starting.
I20240503 08:18:07.273700 25574 tablet_bootstrap.cc:492] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: Bootstrap starting.
I20240503 08:18:07.273818 25572 tablet_bootstrap.cc:654] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:07.274187 25572 log.cc:824] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:07.274295 25574 tablet_bootstrap.cc:654] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:07.274616 25573 tablet_bootstrap.cc:492] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24: Bootstrap starting.
I20240503 08:18:07.274619 25574 log.cc:824] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:07.274971 25572 tablet_bootstrap.cc:492] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e: No bootstrap required, opened a new log
I20240503 08:18:07.275050 25572 ts_tablet_manager.cc:1367] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:18:07.275175 25573 tablet_bootstrap.cc:654] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:07.275434 25573 log.cc:824] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:07.275508 25574 tablet_bootstrap.cc:492] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: No bootstrap required, opened a new log
I20240503 08:18:07.275570 25574 ts_tablet_manager.cc:1367] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:18:07.276089 25573 tablet_bootstrap.cc:492] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24: No bootstrap required, opened a new log
I20240503 08:18:07.276152 25573 ts_tablet_manager.cc:1367] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:18:07.277621 25572 raft_consensus.cc:348] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0684b1ed7146e882af6d1059291d24" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43241 } } peers { permanent_uuid: "34ed4ae82cca47a7a409e0380575170e" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38291 } } peers { permanent_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 43953 } }
I20240503 08:18:07.277791 25572 raft_consensus.cc:374] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:07.277817 25572 raft_consensus.cc:729] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ed4ae82cca47a7a409e0380575170e, State: Initialized, Role: FOLLOWER
I20240503 08:18:07.277940 25572 consensus_queue.cc:260] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0684b1ed7146e882af6d1059291d24" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43241 } } peers { permanent_uuid: "34ed4ae82cca47a7a409e0380575170e" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38291 } } peers { permanent_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 43953 } }
I20240503 08:18:07.278004 25573 raft_consensus.cc:348] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0684b1ed7146e882af6d1059291d24" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43241 } } peers { permanent_uuid: "34ed4ae82cca47a7a409e0380575170e" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38291 } } peers { permanent_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 43953 } }
I20240503 08:18:07.278148 25573 raft_consensus.cc:374] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:07.278187 25573 raft_consensus.cc:729] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0684b1ed7146e882af6d1059291d24, State: Initialized, Role: FOLLOWER
I20240503 08:18:07.278263 25572 ts_tablet_manager.cc:1398] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 08:18:07.278287 25573 consensus_queue.cc:260] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0684b1ed7146e882af6d1059291d24" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43241 } } peers { permanent_uuid: "34ed4ae82cca47a7a409e0380575170e" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38291 } } peers { permanent_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 43953 } }
I20240503 08:18:07.278410 25296 heartbeater.cc:503] Master 127.6.68.254:34503 was elected leader, sending a full tablet report...
I20240503 08:18:07.278331 25574 raft_consensus.cc:348] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0684b1ed7146e882af6d1059291d24" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43241 } } peers { permanent_uuid: "34ed4ae82cca47a7a409e0380575170e" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38291 } } peers { permanent_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 43953 } }
I20240503 08:18:07.278450 25574 raft_consensus.cc:374] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:07.278482 25574 raft_consensus.cc:729] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cdcbe62c1ee41fb8e016f2ffb8c6b13, State: Initialized, Role: FOLLOWER
I20240503 08:18:07.278544 25573 ts_tablet_manager.cc:1398] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 08:18:07.278566 25574 consensus_queue.cc:260] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0684b1ed7146e882af6d1059291d24" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43241 } } peers { permanent_uuid: "34ed4ae82cca47a7a409e0380575170e" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38291 } } peers { permanent_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 43953 } }
I20240503 08:18:07.278671 25558 heartbeater.cc:503] Master 127.6.68.254:34503 was elected leader, sending a full tablet report...
I20240503 08:18:07.278793 25574 ts_tablet_manager.cc:1398] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:18:07.278901 25427 heartbeater.cc:503] Master 127.6.68.254:34503 was elected leader, sending a full tablet report...
I20240503 08:18:07.347808 25579 raft_consensus.cc:482] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:18:07.347936 25579 raft_consensus.cc:504] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0684b1ed7146e882af6d1059291d24" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43241 } } peers { permanent_uuid: "34ed4ae82cca47a7a409e0380575170e" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38291 } } peers { permanent_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 43953 } }
I20240503 08:18:07.348270 25579 leader_election.cc:288] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34ed4ae82cca47a7a409e0380575170e (127.6.68.193:38291), 8cdcbe62c1ee41fb8e016f2ffb8c6b13 (127.6.68.194:43953)
I20240503 08:18:07.350325 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.348421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:43953 (local address 127.6.68.195:39597)
0503 08:18:07.348479 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:07.348491 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:07.348535 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:07.348742 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:07.348743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:07.348745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:07.348910 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.348911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.349550 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.349551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:07.350115 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.350116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.350185 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.350189 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:18:07.350221 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:18:07.350254 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20240503 08:18:07.350441 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.348421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:38291 (local address 127.6.68.195:46799)
0503 08:18:07.348634 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:07.348640 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:07.348685 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:07.348808 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:07.348809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:07.348810 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:07.349010 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.349011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.349628 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.349629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:07.350206 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:07.350207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.350286 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.350288 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:18:07.350321 (+ 33us) client_negotiation.cc:240] Negotiation successful
0503 08:18:07.350353 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":10,"thread_start_us":148,"threads_started":1}
I20240503 08:18:07.350783 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.348555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:39597 (local address 127.6.68.194:43953)
0503 08:18:07.348605 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:07.348607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:07.348615 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:18:07.348644 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:07.348645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:07.348656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:07.348733 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:07.349000 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.349467 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.350204 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.350687 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.350694 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:07.350714 (+ 20us) server_negotiation.cc:299] Negotiation successful
0503 08:18:07.350736 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240503 08:18:07.350852 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.348508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:46799 (local address 127.6.68.193:38291)
0503 08:18:07.348553 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:07.348555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:07.348701 (+ 146us) server_negotiation.cc:407] Connection header received
0503 08:18:07.348735 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:07.348737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:07.348751 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:07.348802 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:07.349074 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.349541 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.350283 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.350743 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.350750 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:07.350772 (+ 22us) server_negotiation.cc:299] Negotiation successful
0503 08:18:07.350800 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:18:07.351096 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3587153e23684f8f9ef5a0f8942cca62" candidate_uuid: "1b0684b1ed7146e882af6d1059291d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" is_pre_election: true
I20240503 08:18:07.351161 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3587153e23684f8f9ef5a0f8942cca62" candidate_uuid: "1b0684b1ed7146e882af6d1059291d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ed4ae82cca47a7a409e0380575170e" is_pre_election: true
I20240503 08:18:07.351261 25250 raft_consensus.cc:2454] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0684b1ed7146e882af6d1059291d24 in term 0.
I20240503 08:18:07.351261 25381 raft_consensus.cc:2454] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0684b1ed7146e882af6d1059291d24 in term 0.
I20240503 08:18:07.351496 25448 leader_election.cc:302] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0684b1ed7146e882af6d1059291d24, 34ed4ae82cca47a7a409e0380575170e; no voters:
I20240503 08:18:07.351673 25579 raft_consensus.cc:2781] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:18:07.351734 25579 raft_consensus.cc:482] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:18:07.351761 25579 raft_consensus.cc:3037] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:07.352375 25579 raft_consensus.cc:504] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0684b1ed7146e882af6d1059291d24" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43241 } } peers { permanent_uuid: "34ed4ae82cca47a7a409e0380575170e" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38291 } } peers { permanent_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 43953 } }
I20240503 08:18:07.352499 25579 leader_election.cc:288] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [CANDIDATE]: Term 1 election: Requested vote from peers 34ed4ae82cca47a7a409e0380575170e (127.6.68.193:38291), 8cdcbe62c1ee41fb8e016f2ffb8c6b13 (127.6.68.194:43953)
I20240503 08:18:07.352658 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3587153e23684f8f9ef5a0f8942cca62" candidate_uuid: "1b0684b1ed7146e882af6d1059291d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ed4ae82cca47a7a409e0380575170e"
I20240503 08:18:07.352705 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3587153e23684f8f9ef5a0f8942cca62" candidate_uuid: "1b0684b1ed7146e882af6d1059291d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13"
I20240503 08:18:07.352743 25250 raft_consensus.cc:3037] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:07.352774 25381 raft_consensus.cc:3037] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:07.353649 25381 raft_consensus.cc:2454] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0684b1ed7146e882af6d1059291d24 in term 1.
I20240503 08:18:07.353649 25250 raft_consensus.cc:2454] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0684b1ed7146e882af6d1059291d24 in term 1.
I20240503 08:18:07.353881 25447 leader_election.cc:302] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0684b1ed7146e882af6d1059291d24, 8cdcbe62c1ee41fb8e016f2ffb8c6b13; no voters:
I20240503 08:18:07.354002 25579 raft_consensus.cc:2781] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:07.354292 25579 raft_consensus.cc:686] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 1 LEADER]: Becoming Leader. State: Replica: 1b0684b1ed7146e882af6d1059291d24, State: Running, Role: LEADER
I20240503 08:18:07.354463 25579 consensus_queue.cc:237] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0684b1ed7146e882af6d1059291d24" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43241 } } peers { permanent_uuid: "34ed4ae82cca47a7a409e0380575170e" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38291 } } peers { permanent_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 43953 } }
I20240503 08:18:07.355372 25107 catalog_manager.cc:5484] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b0684b1ed7146e882af6d1059291d24 (127.6.68.195). New cstate: current_term: 1 leader_uuid: "1b0684b1ed7146e882af6d1059291d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b0684b1ed7146e882af6d1059291d24" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 43241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34ed4ae82cca47a7a409e0380575170e" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 43953 } health_report { overall_health: UNKNOWN } } }
W20240503 08:18:07.363258 25428 tablet.cc:2369] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:18:07.366276 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.364292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.6.68.195:43241)
0503 08:18:07.364342 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:07.364344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:07.364351 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:07.364390 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:07.364390 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:07.364406 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:07.364470 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:07.364688 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.365189 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:07.365757 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:07.365870 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:07.366000 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:07.366072 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:07.366130 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:07.366207 (+ 77us) server_negotiation.cc:299] Negotiation successful
0503 08:18:07.366217 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240503 08:18:07.367941 25381 raft_consensus.cc:1261] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13 [term 1 FOLLOWER]: Refusing update from remote peer 1b0684b1ed7146e882af6d1059291d24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:07.367941 25250 raft_consensus.cc:1261] T 3587153e23684f8f9ef5a0f8942cca62 P 34ed4ae82cca47a7a409e0380575170e [term 1 FOLLOWER]: Refusing update from remote peer 1b0684b1ed7146e882af6d1059291d24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:07.368294 25582 consensus_queue.cc:1035] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cdcbe62c1ee41fb8e016f2ffb8c6b13" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 43953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:07.368433 25579 consensus_queue.cc:1035] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34ed4ae82cca47a7a409e0380575170e" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:07.370445 25585 mvcc.cc:205] Tried to move back new op lower bound from 7023510681056604160 to 7023510681004568576. Current Snapshot: MvccSnapshot[applied={T|T < 7023510681056604160}]
I20240503 08:18:07.371199 25586 mvcc.cc:205] Tried to move back new op lower bound from 7023510681056604160 to 7023510681004568576. Current Snapshot: MvccSnapshot[applied={T|T < 7023510681056604160 or (T in {7023510681056604160})}]
I20240503 08:18:07.372920 25108 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49588:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:07.373028 25108 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49588:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:07.373770 25108 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b5f930cd65149b4add2f30907142c31: Sending DeleteTablet for 3 replicas of tablet 3587153e23684f8f9ef5a0f8942cca62
I20240503 08:18:07.374233 25361 tablet_service.cc:1514] Processing DeleteTablet for tablet 3587153e23684f8f9ef5a0f8942cca62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:07 UTC) from {username='slave'} at 127.0.0.1:38024
I20240503 08:18:07.374239 25230 tablet_service.cc:1514] Processing DeleteTablet for tablet 3587153e23684f8f9ef5a0f8942cca62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:07 UTC) from {username='slave'} at 127.0.0.1:42116
I20240503 08:18:07.374436 25492 tablet_service.cc:1514] Processing DeleteTablet for tablet 3587153e23684f8f9ef5a0f8942cca62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:07 UTC) from {username='slave'} at 127.0.0.1:47038
I20240503 08:18:07.374599 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 25168
I20240503 08:18:07.374631 25596 tablet_replica.cc:330] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24: stopping tablet replica
I20240503 08:18:07.374704 25596 raft_consensus.cc:2229] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:18:07.374787 25596 raft_consensus.cc:2258] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:07.374915 25594 tablet_replica.cc:330] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: stopping tablet replica
I20240503 08:18:07.374985 25594 raft_consensus.cc:2229] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:18:07.375077 25594 raft_consensus.cc:2258] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:07.375566 25596 ts_tablet_manager.cc:1875] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:07.377645 25596 ts_tablet_manager.cc:1888] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:18:07.377710 25596 log.cc:1196] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/3587153e23684f8f9ef5a0f8942cca62
I20240503 08:18:07.377998 25596 ts_tablet_manager.cc:1909] T 3587153e23684f8f9ef5a0f8942cca62 P 1b0684b1ed7146e882af6d1059291d24: Deleting consensus metadata
I20240503 08:18:07.378353 25594 ts_tablet_manager.cc:1875] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:07.378528 25093 catalog_manager.cc:4830] TS 1b0684b1ed7146e882af6d1059291d24 (127.6.68.195:43241): tablet 3587153e23684f8f9ef5a0f8942cca62 (table test-table [id=33360a4d40564ba18f9d86b1411d6584]) successfully deleted
I20240503 08:18:07.380282 25594 ts_tablet_manager.cc:1888] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:18:07.380360 25594 log.cc:1196] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/3587153e23684f8f9ef5a0f8942cca62
I20240503 08:18:07.380636 25594 ts_tablet_manager.cc:1909] T 3587153e23684f8f9ef5a0f8942cca62 P 8cdcbe62c1ee41fb8e016f2ffb8c6b13: Deleting consensus metadata
I20240503 08:18:07.381095 25094 catalog_manager.cc:4830] TS 8cdcbe62c1ee41fb8e016f2ffb8c6b13 (127.6.68.194:43953): tablet 3587153e23684f8f9ef5a0f8942cca62 (table test-table [id=33360a4d40564ba18f9d86b1411d6584]) successfully deleted
W20240503 08:18:07.383067 25095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:38291 (error 108)
I20240503 08:18:07.383550 25568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:07.383412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:38291 (local address 127.0.0.1:42120)
0503 08:18:07.383469 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:07.383507 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:38291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20240503 08:18:07.383646 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 25299
W20240503 08:18:07.383682 25095 catalog_manager.cc:4573] TS 34ed4ae82cca47a7a409e0380575170e (127.6.68.193:38291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3587153e23684f8f9ef5a0f8942cca62: Network error: Client connection negotiation failed: client connection to 127.6.68.193:38291: connect: Connection refused (error 111)
I20240503 08:18:07.389185 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 25430
I20240503 08:18:07.393702 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 25077
2024-05-03T08:18:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (729 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:07 dist-test-slave-3g67 krb5kdc[25600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:07 dist-test-slave-3g67 krb5kdc[25600](info): set up 2 sockets
May 03 08:18:07 dist-test-slave-3g67 krb5kdc[25600](info): commencing operation
krb5kdc: starting...
W20240503 08:18:08.359879 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:08 dist-test-slave-3g67 krb5kdc[25600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:18:08.478871 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34595
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:34595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:08.557539 25613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:08.557742 25613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:08.557780 25613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:08.559093 25613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:08.559149 25613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:08.559170 25613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:08.559191 25613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:08.559212 25613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:08.560863 25613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:34595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:34595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:08.561115 25613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:08.561323 25613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:08.566049 25618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:08.566637 25621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:08.567276 25619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:08.567857 25613 server_base.cc:1034] running on GCE node
I20240503 08:18:08.568122 25613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:08.568370 25613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:08.569523 25613 hybrid_clock.cc:648] HybridClock initialized: now 1714724288569490 us; error 48 us; skew 500 ppm
May 03 08:18:08 dist-test-slave-3g67 krb5kdc[25600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724288, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:08.572165 25613 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:18:08.572609 25613 webserver.cc:460] Webserver started at http://127.6.68.254:41023/ using document root <none> and password file <none>
I20240503 08:18:08.572801 25613 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:08.572847 25613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:08.572984 25613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:08.574012 25613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "a406bfb139bf471c9c1bb49da08f03a0"
format_stamp: "Formatted at 2024-05-03 08:18:08 on dist-test-slave-3g67"
server_key: "e97db091485eb1f81ed2d39d6aa04b1c"
server_key_iv: "c13bdbf84d0b420f3fc4914c8c3689d8"
server_key_version: "encryptionkey@0"
I20240503 08:18:08.574324 25613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "a406bfb139bf471c9c1bb49da08f03a0"
format_stamp: "Formatted at 2024-05-03 08:18:08 on dist-test-slave-3g67"
server_key: "e97db091485eb1f81ed2d39d6aa04b1c"
server_key_iv: "c13bdbf84d0b420f3fc4914c8c3689d8"
server_key_version: "encryptionkey@0"
I20240503 08:18:08.575608 25613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240503 08:18:08.576339 25628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:08.576547 25613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:18:08.576625 25613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "a406bfb139bf471c9c1bb49da08f03a0"
format_stamp: "Formatted at 2024-05-03 08:18:08 on dist-test-slave-3g67"
server_key: "e97db091485eb1f81ed2d39d6aa04b1c"
server_key_iv: "c13bdbf84d0b420f3fc4914c8c3689d8"
server_key_version: "encryptionkey@0"
I20240503 08:18:08.576689 25613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:08.607750 25613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:08.609752 25613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:08.609880 25613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:08.613705 25613 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:34595
I20240503 08:18:08.613760 25680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:34595 every 8 connection(s)
I20240503 08:18:08.614060 25613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:08.614881 25681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:08.617024 25681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0: Bootstrap starting.
I20240503 08:18:08.617574 25681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:08.617830 25681 log.cc:824] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:08.618486 25681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0: No bootstrap required, opened a new log
I20240503 08:18:08.619904 25681 raft_consensus.cc:348] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a406bfb139bf471c9c1bb49da08f03a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34595 } }
I20240503 08:18:08.620034 25681 raft_consensus.cc:374] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:08.620051 25681 raft_consensus.cc:729] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a406bfb139bf471c9c1bb49da08f03a0, State: Initialized, Role: FOLLOWER
I20240503 08:18:08.620169 25681 consensus_queue.cc:260] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a406bfb139bf471c9c1bb49da08f03a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34595 } }
I20240503 08:18:08.620239 25681 raft_consensus.cc:388] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:08.620260 25681 raft_consensus.cc:482] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:08.620307 25681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:08.620836 25681 raft_consensus.cc:504] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a406bfb139bf471c9c1bb49da08f03a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34595 } }
I20240503 08:18:08.620929 25681 leader_election.cc:302] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a406bfb139bf471c9c1bb49da08f03a0; no voters:
I20240503 08:18:08.621085 25681 leader_election.cc:288] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:08.621145 25684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:08.621317 25684 raft_consensus.cc:686] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [term 1 LEADER]: Becoming Leader. State: Replica: a406bfb139bf471c9c1bb49da08f03a0, State: Running, Role: LEADER
I20240503 08:18:08.621410 25684 consensus_queue.cc:237] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a406bfb139bf471c9c1bb49da08f03a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34595 } }
I20240503 08:18:08.621667 25681 sys_catalog.cc:564] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:08.621788 25685 sys_catalog.cc:455] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a406bfb139bf471c9c1bb49da08f03a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a406bfb139bf471c9c1bb49da08f03a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34595 } } }
I20240503 08:18:08.621902 25685 sys_catalog.cc:458] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:08.621798 25686 sys_catalog.cc:455] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a406bfb139bf471c9c1bb49da08f03a0. Latest consensus state: current_term: 1 leader_uuid: "a406bfb139bf471c9c1bb49da08f03a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a406bfb139bf471c9c1bb49da08f03a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 34595 } } }
I20240503 08:18:08.621981 25686 sys_catalog.cc:458] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:08.622282 25688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:08.622761 25688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:08.623818 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 25613
I20240503 08:18:08.623926 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:08.624114 6419 external_mini_cluster.cc:1428] Setting key c3579abb62749bd234f8f9b7408a6136
I20240503 08:18:08.624171 25688 catalog_manager.cc:1347] Generated new cluster ID: a0202f9d4d894ba5941699f42489bf42
I20240503 08:18:08.624222 25688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 03 08:18:08 dist-test-slave-3g67 krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:08.637814 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:08.628005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.6.68.254:34595)
0503 08:18:08.628200 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:08.628212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:08.628236 (+ 24us) server_negotiation.cc:407] Connection header received
0503 08:18:08.628343 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:08.628348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:08.628395 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:08.628532 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:08.629495 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:08.630659 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:08.631505 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:08.631776 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:08.634290 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:08.634298 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:08.634298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:08.634319 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:08.636439 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:08.636788 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:08.636790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:08.636791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:08.636848 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:08.636994 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:08.636995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:08.636996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:08.637240 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:08.637391 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:08.637463 (+ 72us) server_negotiation.cc:299] Negotiation successful
0503 08:18:08.637592 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
I20240503 08:18:08.652449 25688 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:08.653072 25688 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:08.661972 25688 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0: Generated new TSK 0
I20240503 08:18:08.662165 25688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.68.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
Entry for principal kudu/127.6.68.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab.
I20240503 08:18:08.691512 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34595
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:08.775012 25707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:08.775223 25707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:08.775264 25707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:08.776691 25707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:08.776738 25707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:08.776786 25707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:18:08.778455 25707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.193.keytab
--principal=kudu/127.6.68.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:08.778697 25707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:08.778887 25707 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:08.783768 25713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:08.783969 25715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:08.784651 25712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:08.785173 25707 server_base.cc:1034] running on GCE node
I20240503 08:18:08.785369 25707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:08.785583 25707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:08.786736 25707 hybrid_clock.cc:648] HybridClock initialized: now 1714724288786721 us; error 33 us; skew 500 ppm
May 03 08:18:08 dist-test-slave-3g67 krb5kdc[25600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724288, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:08.789208 25707 init.cc:377] Logged in from keytab as kudu/127.6.68.193@KRBTEST.COM (short username kudu)
I20240503 08:18:08.789618 25707 webserver.cc:460] Webserver started at http://127.6.68.193:36045/ using document root <none> and password file <none>
I20240503 08:18:08.789798 25707 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:08.789841 25707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:08.789949 25707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:08.790946 25707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "a97eb980747c441193a804363e3fe1db"
format_stamp: "Formatted at 2024-05-03 08:18:08 on dist-test-slave-3g67"
server_key: "dc186ecf2e160cdd065d3375e7001344"
server_key_iv: "8a98ca000f110acf93151d216280b100"
server_key_version: "encryptionkey@0"
I20240503 08:18:08.791246 25707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "a97eb980747c441193a804363e3fe1db"
format_stamp: "Formatted at 2024-05-03 08:18:08 on dist-test-slave-3g67"
server_key: "dc186ecf2e160cdd065d3375e7001344"
server_key_iv: "8a98ca000f110acf93151d216280b100"
server_key_version: "encryptionkey@0"
I20240503 08:18:08.792541 25707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:18:08.793222 25722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:08.793392 25707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:08.793468 25707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "a97eb980747c441193a804363e3fe1db"
format_stamp: "Formatted at 2024-05-03 08:18:08 on dist-test-slave-3g67"
server_key: "dc186ecf2e160cdd065d3375e7001344"
server_key_iv: "8a98ca000f110acf93151d216280b100"
server_key_version: "encryptionkey@0"
I20240503 08:18:08.793535 25707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:08.812326 25707 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:08.814224 25707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:08.814349 25707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:08.814563 25707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:08.814872 25707 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:08.814904 25707 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:08.814940 25707 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:08.814966 25707 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:08.820150 25707 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:39901
I20240503 08:18:08.820219 25835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:39901 every 8 connection(s)
I20240503 08:18:08.820500 25707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
May 03 08:18:08 dist-test-slave-3g67 krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724288, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.193@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:08.826907 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 25707
I20240503 08:18:08.827051 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:18:08.827263 6419 external_mini_cluster.cc:1428] Setting key f63244e5043c26f72c77195fcd2a396e
I20240503 08:18:08.829744 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:08.821350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:41305 (local address 127.6.68.254:34595)
0503 08:18:08.821446 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:08.821449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:08.821592 (+ 143us) server_negotiation.cc:407] Connection header received
0503 08:18:08.822154 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:08.822156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:08.822171 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:08.822219 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:08.823038 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:08.823602 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:08.824544 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:08.824725 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:08.826656 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:08.826664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:08.826664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:08.826671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:08.828363 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:08.828841 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:08.828841 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:08.828842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:08.828859 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:08.829049 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:08.829050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:08.829050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:08.829133 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:08.829177 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:08.829480 (+ 303us) server_negotiation.cc:299] Negotiation successful
0503 08:18:08.829603 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20240503 08:18:08.829782 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:08.821282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34595 (local address 127.6.68.193:41305)
0503 08:18:08.821478 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:08.821509 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:08.822031 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:08.822279 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:08.822285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:08.822603 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:08.822935 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:08.822939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:08.823704 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:08.823705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:08.824455 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:08.824457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:08.824594 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:08.825166 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:08.825183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:08.826544 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:08.828445 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:08.828453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:08.828456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:08.828768 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:08.828911 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:08.828912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:08.828913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:08.828998 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:08.829181 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:08.829185 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:08.829333 (+ 148us) client_negotiation.cc:769] Sending connection context
0503 08:18:08.829458 (+ 125us) client_negotiation.cc:240] Negotiation successful
0503 08:18:08.829626 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20240503 08:18:08.830291 25836 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34595
I20240503 08:18:08.830399 25836 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:08.830642 25836 heartbeater.cc:511] Master 127.6.68.254:34595 requested a full tablet report, sending...
I20240503 08:18:08.831130 25645 ts_manager.cc:194] Registered new tserver with Master: a97eb980747c441193a804363e3fe1db (127.6.68.193:39901)
I20240503 08:18:08.831835 25645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.193@KRBTEST.COM'} at 127.6.68.193:41305
WARNING: no policy specified for kudu/127.6.68.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
Entry for principal kudu/127.6.68.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab.
I20240503 08:18:08.854934 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34595
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:08.935266 25841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:08.935477 25841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:08.935500 25841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:08.936836 25841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:08.936882 25841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:08.936908 25841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:18:08.938484 25841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.194.keytab
--principal=kudu/127.6.68.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:08.938683 25841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:08.938854 25841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:08.943713 25846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:08.944089 25849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:08.944669 25847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:08.945303 25841 server_base.cc:1034] running on GCE node
I20240503 08:18:08.945503 25841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:08.945719 25841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:08.946857 25841 hybrid_clock.cc:648] HybridClock initialized: now 1714724288946840 us; error 36 us; skew 500 ppm
May 03 08:18:08 dist-test-slave-3g67 krb5kdc[25600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724288, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:08.949389 25841 init.cc:377] Logged in from keytab as kudu/127.6.68.194@KRBTEST.COM (short username kudu)
I20240503 08:18:08.949807 25841 webserver.cc:460] Webserver started at http://127.6.68.194:45345/ using document root <none> and password file <none>
I20240503 08:18:08.949982 25841 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:08.950027 25841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:08.950136 25841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:08.951114 25841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "e1865be0728d4c4bb81748bd63d9a8f4"
format_stamp: "Formatted at 2024-05-03 08:18:08 on dist-test-slave-3g67"
server_key: "28fdcab824bed596b1c663e35b7f1f93"
server_key_iv: "dd43df0daa23c4d9da7877e6d1a0801c"
server_key_version: "encryptionkey@0"
I20240503 08:18:08.951413 25841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "e1865be0728d4c4bb81748bd63d9a8f4"
format_stamp: "Formatted at 2024-05-03 08:18:08 on dist-test-slave-3g67"
server_key: "28fdcab824bed596b1c663e35b7f1f93"
server_key_iv: "dd43df0daa23c4d9da7877e6d1a0801c"
server_key_version: "encryptionkey@0"
I20240503 08:18:08.952693 25841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:18:08.953462 25856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:08.953652 25841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:08.953740 25841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "e1865be0728d4c4bb81748bd63d9a8f4"
format_stamp: "Formatted at 2024-05-03 08:18:08 on dist-test-slave-3g67"
server_key: "28fdcab824bed596b1c663e35b7f1f93"
server_key_iv: "dd43df0daa23c4d9da7877e6d1a0801c"
server_key_version: "encryptionkey@0"
I20240503 08:18:08.953806 25841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:08.966493 25841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:08.968729 25841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:08.968875 25841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:08.969095 25841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:08.969390 25841 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:08.969425 25841 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:08.969460 25841 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:08.969488 25841 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:08.974861 25841 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:33905
I20240503 08:18:08.974915 25969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:33905 every 8 connection(s)
I20240503 08:18:08.975217 25841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:18:08.979660 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 25841
I20240503 08:18:08.979784 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:18:08.979945 6419 external_mini_cluster.cc:1428] Setting key 02d7e0920e94ffbc9bec49c9715535b9
May 03 08:18:08 dist-test-slave-3g67 krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724288, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.194@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:08.984503 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:08.975968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:35043 (local address 127.6.68.254:34595)
0503 08:18:08.976029 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:08.976032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:08.976216 (+ 184us) server_negotiation.cc:407] Connection header received
0503 08:18:08.976718 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:08.976719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:08.976732 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:08.976793 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:08.977571 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:08.978076 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:08.979033 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:08.979227 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:08.981531 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:08.981537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:08.981537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:08.981544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:08.983098 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:08.983586 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:08.983587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:08.983587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:08.983607 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:08.983855 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:08.983856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:08.983856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:08.983952 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:08.984010 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:08.984260 (+ 250us) server_negotiation.cc:299] Negotiation successful
0503 08:18:08.984357 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240503 08:18:08.984607 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:08.975883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34595 (local address 127.6.68.194:35043)
0503 08:18:08.976086 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:08.976129 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:08.976616 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:08.976857 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:08.976861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:08.977163 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:08.977453 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:08.977457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:08.978162 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:08.978163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:08.978943 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:08.978945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:08.979077 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:08.979728 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:08.979744 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:08.981419 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:08.983195 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:08.983200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:08.983203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:08.983510 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:08.983724 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:08.983725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:08.983725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:08.983789 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:08.984008 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:08.984013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:08.984133 (+ 120us) client_negotiation.cc:769] Sending connection context
0503 08:18:08.984232 (+ 99us) client_negotiation.cc:240] Negotiation successful
0503 08:18:08.984396 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
I20240503 08:18:08.985123 25970 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34595
I20240503 08:18:08.985239 25970 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:08.985450 25970 heartbeater.cc:511] Master 127.6.68.254:34595 requested a full tablet report, sending...
I20240503 08:18:08.985878 25645 ts_manager.cc:194] Registered new tserver with Master: e1865be0728d4c4bb81748bd63d9a8f4 (127.6.68.194:33905)
I20240503 08:18:08.986498 25645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.194@KRBTEST.COM'} at 127.6.68.194:35043
WARNING: no policy specified for kudu/127.6.68.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
Entry for principal kudu/127.6.68.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab.
I20240503 08:18:09.007323 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34595
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:09.086973 25975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:09.087179 25975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:09.087214 25975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:09.088598 25975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:09.088652 25975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:09.088693 25975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:18:09.090378 25975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.195.keytab
--principal=kudu/127.6.68.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:34595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:09.090638 25975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:09.090824 25975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:09.095610 25983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:09.096058 25980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:09.096588 25981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:09.097283 25975 server_base.cc:1034] running on GCE node
I20240503 08:18:09.097452 25975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:09.097777 25975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:09.098943 25975 hybrid_clock.cc:648] HybridClock initialized: now 1714724289098926 us; error 39 us; skew 500 ppm
May 03 08:18:09 dist-test-slave-3g67 krb5kdc[25600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724289, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:09.101522 25975 init.cc:377] Logged in from keytab as kudu/127.6.68.195@KRBTEST.COM (short username kudu)
I20240503 08:18:09.101958 25975 webserver.cc:460] Webserver started at http://127.6.68.195:45403/ using document root <none> and password file <none>
I20240503 08:18:09.102144 25975 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:09.102190 25975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:09.102312 25975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:09.103309 25975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "4c5f84145e4f421493cd48f7a4208615"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "932f5e5f6a07b05fe5ea96d89d888d7f"
server_key_iv: "663a480032dad29be4222750f5f8e6b3"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.103653 25975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "4c5f84145e4f421493cd48f7a4208615"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "932f5e5f6a07b05fe5ea96d89d888d7f"
server_key_iv: "663a480032dad29be4222750f5f8e6b3"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.104887 25975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:09.105690 25990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.105899 25975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:09.105975 25975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "4c5f84145e4f421493cd48f7a4208615"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "932f5e5f6a07b05fe5ea96d89d888d7f"
server_key_iv: "663a480032dad29be4222750f5f8e6b3"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.106042 25975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:09.122633 25975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:09.124624 25975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:09.124725 25975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:09.124878 25975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:09.125176 25975 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:09.125211 25975 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.125234 25975 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:09.125262 25975 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.130401 25975 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:44749
I20240503 08:18:09.130452 26103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:44749 every 8 connection(s)
I20240503 08:18:09.130721 25975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:18:09.131843 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 25975
I20240503 08:18:09.131959 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:18:09.132141 6419 external_mini_cluster.cc:1428] Setting key b9057475402d9a75cfc0bcf2b7a2a755
May 03 08:18:09 dist-test-slave-3g67 krb5kdc[25600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724289, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.195@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:09.139567 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.131433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:50983 (local address 127.6.68.254:34595)
0503 08:18:09.131501 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.131504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.131685 (+ 181us) server_negotiation.cc:407] Connection header received
0503 08:18:09.132360 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.132361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.132374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.132433 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:09.133339 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.133833 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.134802 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.134921 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.136905 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:09.136909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:09.136909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:09.136916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:09.138358 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:09.138793 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:09.138795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:09.138795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:09.138818 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:09.138977 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:09.138978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:09.138978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:09.139088 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:09.139138 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.139417 (+ 279us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.139496 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:18:09.139781 26106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.131400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:34595 (local address 127.6.68.195:50983)
0503 08:18:09.131556 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.131588 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.132242 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.132489 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.132494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.132922 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:09.133259 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.133263 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.133931 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.133932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.134723 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.134725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.134837 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.135409 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:09.135425 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:09.136820 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:09.138488 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:09.138493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:09.138496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:09.138701 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:09.138876 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:09.138876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:09.138877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:09.138926 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:09.139166 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:09.139172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:09.139295 (+ 123us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.139401 (+ 106us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.139541 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20240503 08:18:09.140420 26104 heartbeater.cc:347] Connected to a master server at 127.6.68.254:34595
I20240503 08:18:09.140537 26104 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:09.140733 26104 heartbeater.cc:511] Master 127.6.68.254:34595 requested a full tablet report, sending...
I20240503 08:18:09.141094 25645 ts_manager.cc:194] Registered new tserver with Master: 4c5f84145e4f421493cd48f7a4208615 (127.6.68.195:44749)
I20240503 08:18:09.141494 25645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.68.195@KRBTEST.COM'} at 127.6.68.195:50983
I20240503 08:18:09.143821 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:18:09.150105 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.145038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.6.68.254:34595)
0503 08:18:09.145093 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.145095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.145173 (+ 78us) server_negotiation.cc:407] Connection header received
0503 08:18:09.145228 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.145228 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.145236 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.145286 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:09.145930 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.146383 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.146887 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.146987 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.147996 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:09.147999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:09.147999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:09.148005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:09.149313 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:09.149604 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:09.149605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:09.149605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:09.149621 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:09.149723 (+ 102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:09.149723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:09.149723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:09.149828 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:09.149899 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.149974 (+ 75us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.150036 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240503 08:18:09.151813 25645 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:18:09.153229 25645 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:18:09.158813 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.155340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:33905 (local address 127.0.0.1:36796)
0503 08:18:09.155955 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.155965 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.156020 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.156546 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.156546 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.156548 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:09.156725 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.156726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.157728 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.157730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.158533 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.158535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.158600 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.158639 (+ 39us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.158691 (+ 52us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.158733 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":63,"threads_started":1}
I20240503 08:18:09.158982 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.155340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39901 (local address 127.0.0.1:46808)
0503 08:18:09.156086 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.156094 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.156136 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.156459 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.156460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.156462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:09.156705 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.156706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.157945 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.157946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.158752 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.158754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.158821 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.158825 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.158857 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.158897 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":11,"thread_start_us":54,"threads_started":1}
I20240503 08:18:09.159360 26116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.155612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.6.68.194:33905)
0503 08:18:09.155808 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.155815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.156391 (+ 576us) server_negotiation.cc:407] Connection header received
0503 08:18:09.156431 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.156433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.156451 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.156800 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:09.156872 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.157659 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.158635 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.159190 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.159240 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.159269 (+ 29us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.159302 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":114,"threads_started":1}
I20240503 08:18:09.159723 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.155449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.6.68.195:44749)
0503 08:18:09.155647 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.155656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.155822 (+ 166us) server_negotiation.cc:407] Connection header received
0503 08:18:09.155919 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.155921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.155941 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.156018 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:09.156444 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.157588 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.158652 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.159203 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.159246 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.159571 (+ 325us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.159644 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20240503 08:18:09.159870 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.155710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.6.68.193:39901)
0503 08:18:09.155878 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.155884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.156176 (+ 292us) server_negotiation.cc:407] Connection header received
0503 08:18:09.156218 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.156221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.156236 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.156376 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:09.156772 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.157872 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.159157 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.159671 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.159709 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.159739 (+ 30us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.159788 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20240503 08:18:09.159960 26114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.155331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:44749 (local address 127.0.0.1:36464)
0503 08:18:09.155708 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.155737 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.155828 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.156043 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.156046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.156048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:09.156303 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.156305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.157686 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.157687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.158536 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.158538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.159447 (+ 909us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.159451 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.159842 (+ 391us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.159889 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":79,"threads_started":1}
I20240503 08:18:09.159931 25904 tablet_service.cc:1467] Processing CreateTablet for tablet 42ec717dbb704dac8d48ea9998723d61 (DEFAULT_TABLE table=test-table [id=d11702a6c5bd4f9dbb893cc7d40624b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:09.160254 25904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ec717dbb704dac8d48ea9998723d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:09.160375 25770 tablet_service.cc:1467] Processing CreateTablet for tablet 42ec717dbb704dac8d48ea9998723d61 (DEFAULT_TABLE table=test-table [id=d11702a6c5bd4f9dbb893cc7d40624b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:09.160411 26038 tablet_service.cc:1467] Processing CreateTablet for tablet 42ec717dbb704dac8d48ea9998723d61 (DEFAULT_TABLE table=test-table [id=d11702a6c5bd4f9dbb893cc7d40624b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:09.160638 26038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ec717dbb704dac8d48ea9998723d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:09.160638 25770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ec717dbb704dac8d48ea9998723d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:09.162768 26118 tablet_bootstrap.cc:492] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: Bootstrap starting.
I20240503 08:18:09.163426 26118 tablet_bootstrap.cc:654] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:09.163790 26119 tablet_bootstrap.cc:492] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615: Bootstrap starting.
I20240503 08:18:09.163829 26118 log.cc:824] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:09.164294 26119 tablet_bootstrap.cc:654] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:09.164548 26118 tablet_bootstrap.cc:492] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: No bootstrap required, opened a new log
I20240503 08:18:09.164595 26119 log.cc:824] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:09.164621 26118 ts_tablet_manager.cc:1367] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:18:09.164920 26120 tablet_bootstrap.cc:492] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db: Bootstrap starting.
I20240503 08:18:09.165284 26119 tablet_bootstrap.cc:492] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615: No bootstrap required, opened a new log
I20240503 08:18:09.165346 26119 ts_tablet_manager.cc:1367] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:18:09.165549 26120 tablet_bootstrap.cc:654] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:09.165851 26120 log.cc:824] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:09.166646 26120 tablet_bootstrap.cc:492] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db: No bootstrap required, opened a new log
I20240503 08:18:09.166718 26120 ts_tablet_manager.cc:1367] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:18:09.167034 26118 raft_consensus.cc:348] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5f84145e4f421493cd48f7a4208615" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44749 } } peers { permanent_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 33905 } } peers { permanent_uuid: "a97eb980747c441193a804363e3fe1db" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39901 } }
I20240503 08:18:09.167199 26118 raft_consensus.cc:374] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:09.167229 26118 raft_consensus.cc:729] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1865be0728d4c4bb81748bd63d9a8f4, State: Initialized, Role: FOLLOWER
I20240503 08:18:09.167366 26118 consensus_queue.cc:260] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5f84145e4f421493cd48f7a4208615" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44749 } } peers { permanent_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 33905 } } peers { permanent_uuid: "a97eb980747c441193a804363e3fe1db" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39901 } }
I20240503 08:18:09.167429 26119 raft_consensus.cc:348] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5f84145e4f421493cd48f7a4208615" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44749 } } peers { permanent_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 33905 } } peers { permanent_uuid: "a97eb980747c441193a804363e3fe1db" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39901 } }
I20240503 08:18:09.167584 26119 raft_consensus.cc:374] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:09.167639 26119 raft_consensus.cc:729] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5f84145e4f421493cd48f7a4208615, State: Initialized, Role: FOLLOWER
I20240503 08:18:09.167644 26118 ts_tablet_manager.cc:1398] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:18:09.167764 25970 heartbeater.cc:503] Master 127.6.68.254:34595 was elected leader, sending a full tablet report...
I20240503 08:18:09.167745 26119 consensus_queue.cc:260] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5f84145e4f421493cd48f7a4208615" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44749 } } peers { permanent_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 33905 } } peers { permanent_uuid: "a97eb980747c441193a804363e3fe1db" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39901 } }
I20240503 08:18:09.167981 26119 ts_tablet_manager.cc:1398] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:18:09.168020 26104 heartbeater.cc:503] Master 127.6.68.254:34595 was elected leader, sending a full tablet report...
I20240503 08:18:09.169580 26120 raft_consensus.cc:348] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5f84145e4f421493cd48f7a4208615" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44749 } } peers { permanent_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 33905 } } peers { permanent_uuid: "a97eb980747c441193a804363e3fe1db" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39901 } }
I20240503 08:18:09.169692 26120 raft_consensus.cc:374] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:09.169708 26120 raft_consensus.cc:729] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97eb980747c441193a804363e3fe1db, State: Initialized, Role: FOLLOWER
I20240503 08:18:09.169785 26120 consensus_queue.cc:260] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5f84145e4f421493cd48f7a4208615" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44749 } } peers { permanent_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 33905 } } peers { permanent_uuid: "a97eb980747c441193a804363e3fe1db" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39901 } }
I20240503 08:18:09.169997 26120 ts_tablet_manager.cc:1398] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:18:09.170069 25836 heartbeater.cc:503] Master 127.6.68.254:34595 was elected leader, sending a full tablet report...
W20240503 08:18:09.225749 25971 tablet.cc:2369] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:18:09.280444 26126 raft_consensus.cc:482] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:18:09.280629 26126 raft_consensus.cc:504] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5f84145e4f421493cd48f7a4208615" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44749 } } peers { permanent_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 33905 } } peers { permanent_uuid: "a97eb980747c441193a804363e3fe1db" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39901 } }
I20240503 08:18:09.280979 26126 leader_election.cc:288] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c5f84145e4f421493cd48f7a4208615 (127.6.68.195:44749), e1865be0728d4c4bb81748bd63d9a8f4 (127.6.68.194:33905)
I20240503 08:18:09.283612 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.281234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:33905 (local address 127.6.68.193:49647)
0503 08:18:09.281307 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.281321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.281389 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.281598 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.281600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.281603 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:09.281792 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.281793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.282611 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.282612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.283370 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.283372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.283453 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.283457 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.283499 (+ 42us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.283533 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20240503 08:18:09.284102 26116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.281375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:49647 (local address 127.6.68.194:33905)
0503 08:18:09.281424 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.281426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.281436 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:18:09.281502 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.281503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.281512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.281593 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:09.281870 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.282531 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.283487 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.283996 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.284004 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.284025 (+ 21us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.284048 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240503 08:18:09.284103 26127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.281237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:44749 (local address 127.6.68.193:42307)
0503 08:18:09.281478 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.281509 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.281554 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.282001 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.282002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.282004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:09.282201 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.282202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.282977 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.282978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.283881 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.283884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.283948 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.283952 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.283991 (+ 39us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.284028 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":62,"thread_start_us":117,"threads_started":1}
I20240503 08:18:09.284462 25924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ec717dbb704dac8d48ea9998723d61" candidate_uuid: "a97eb980747c441193a804363e3fe1db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" is_pre_election: true
I20240503 08:18:09.284590 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.281363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:42307 (local address 127.6.68.195:44749)
0503 08:18:09.281422 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.281425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.281870 (+ 445us) server_negotiation.cc:407] Connection header received
0503 08:18:09.281900 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.281901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.281912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.281976 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:09.282280 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.282858 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.283967 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.284431 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.284439 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.284463 (+ 24us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.284530 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240503 08:18:09.284638 25924 raft_consensus.cc:2454] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97eb980747c441193a804363e3fe1db in term 0.
I20240503 08:18:09.284875 25723 leader_election.cc:302] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97eb980747c441193a804363e3fe1db, e1865be0728d4c4bb81748bd63d9a8f4; no voters:
I20240503 08:18:09.284901 26058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ec717dbb704dac8d48ea9998723d61" candidate_uuid: "a97eb980747c441193a804363e3fe1db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5f84145e4f421493cd48f7a4208615" is_pre_election: true
I20240503 08:18:09.285022 26058 raft_consensus.cc:2454] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97eb980747c441193a804363e3fe1db in term 0.
I20240503 08:18:09.285053 26126 raft_consensus.cc:2781] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:18:09.285159 26126 raft_consensus.cc:482] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:18:09.285193 26126 raft_consensus.cc:3037] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:09.285894 26126 raft_consensus.cc:504] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5f84145e4f421493cd48f7a4208615" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44749 } } peers { permanent_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 33905 } } peers { permanent_uuid: "a97eb980747c441193a804363e3fe1db" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39901 } }
I20240503 08:18:09.286020 26126 leader_election.cc:288] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [CANDIDATE]: Term 1 election: Requested vote from peers 4c5f84145e4f421493cd48f7a4208615 (127.6.68.195:44749), e1865be0728d4c4bb81748bd63d9a8f4 (127.6.68.194:33905)
I20240503 08:18:09.286212 25924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ec717dbb704dac8d48ea9998723d61" candidate_uuid: "a97eb980747c441193a804363e3fe1db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1865be0728d4c4bb81748bd63d9a8f4"
I20240503 08:18:09.286240 26058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ec717dbb704dac8d48ea9998723d61" candidate_uuid: "a97eb980747c441193a804363e3fe1db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5f84145e4f421493cd48f7a4208615"
I20240503 08:18:09.286299 26058 raft_consensus.cc:3037] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:09.286295 25924 raft_consensus.cc:3037] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:09.287063 26058 raft_consensus.cc:2454] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97eb980747c441193a804363e3fe1db in term 1.
I20240503 08:18:09.287164 25924 raft_consensus.cc:2454] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97eb980747c441193a804363e3fe1db in term 1.
I20240503 08:18:09.287232 25726 leader_election.cc:302] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c5f84145e4f421493cd48f7a4208615, a97eb980747c441193a804363e3fe1db; no voters:
I20240503 08:18:09.287340 26126 raft_consensus.cc:2781] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:09.287518 26126 raft_consensus.cc:686] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 1 LEADER]: Becoming Leader. State: Replica: a97eb980747c441193a804363e3fe1db, State: Running, Role: LEADER
I20240503 08:18:09.287611 26126 consensus_queue.cc:237] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5f84145e4f421493cd48f7a4208615" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44749 } } peers { permanent_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 33905 } } peers { permanent_uuid: "a97eb980747c441193a804363e3fe1db" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39901 } }
I20240503 08:18:09.288427 25645 catalog_manager.cc:5484] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db reported cstate change: term changed from 0 to 1, leader changed from <none> to a97eb980747c441193a804363e3fe1db (127.6.68.193). New cstate: current_term: 1 leader_uuid: "a97eb980747c441193a804363e3fe1db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5f84145e4f421493cd48f7a4208615" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 33905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97eb980747c441193a804363e3fe1db" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 39901 } health_report { overall_health: HEALTHY } } }
I20240503 08:18:09.304353 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.302378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.6.68.193:39901)
0503 08:18:09.302444 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.302447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.302455 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:18:09.302498 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.302498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.302514 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.302582 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:09.302812 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.303267 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.303869 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.304003 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.304046 (+ 43us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:09.304122 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:09.304192 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.304245 (+ 53us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.304268 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240503 08:18:09.305943 26058 raft_consensus.cc:1261] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615 [term 1 FOLLOWER]: Refusing update from remote peer a97eb980747c441193a804363e3fe1db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:09.305944 25924 raft_consensus.cc:1261] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4 [term 1 FOLLOWER]: Refusing update from remote peer a97eb980747c441193a804363e3fe1db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:09.306257 26128 consensus_queue.cc:1035] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5f84145e4f421493cd48f7a4208615" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 44749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:09.306385 26126 consensus_queue.cc:1035] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1865be0728d4c4bb81748bd63d9a8f4" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:09.308506 26131 mvcc.cc:205] Tried to move back new op lower bound from 7023510688994680832 to 7023510688922705920. Current Snapshot: MvccSnapshot[applied={T|T < 7023510688994680832}]
I20240503 08:18:09.308678 26133 mvcc.cc:205] Tried to move back new op lower bound from 7023510688994680832 to 7023510688922705920. Current Snapshot: MvccSnapshot[applied={T|T < 7023510688994680832}]
I20240503 08:18:09.308744 26132 mvcc.cc:205] Tried to move back new op lower bound from 7023510688994680832 to 7023510688922705920. Current Snapshot: MvccSnapshot[applied={T|T < 7023510688994680832}]
I20240503 08:18:09.310698 25644 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38832:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:09.310794 25644 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38832:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:09.311502 25644 catalog_manager.cc:5771] T 00000000000000000000000000000000 P a406bfb139bf471c9c1bb49da08f03a0: Sending DeleteTablet for 3 replicas of tablet 42ec717dbb704dac8d48ea9998723d61
I20240503 08:18:09.311834 25770 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ec717dbb704dac8d48ea9998723d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:09 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:46808
I20240503 08:18:09.311834 26038 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ec717dbb704dac8d48ea9998723d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:09 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:36464
I20240503 08:18:09.311832 25904 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ec717dbb704dac8d48ea9998723d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:09 UTC) from {username='kudu', principal='kudu/127.6.68.254@KRBTEST.COM'} at 127.0.0.1:36796
I20240503 08:18:09.312042 26141 tablet_replica.cc:330] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db: stopping tablet replica
I20240503 08:18:09.312121 26141 raft_consensus.cc:2229] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:18:09.312151 26142 tablet_replica.cc:330] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615: stopping tablet replica
I20240503 08:18:09.312220 26141 raft_consensus.cc:2258] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:09.312484 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 25707
I20240503 08:18:09.312615 26141 ts_tablet_manager.cc:1875] T 42ec717dbb704dac8d48ea9998723d61 P a97eb980747c441193a804363e3fe1db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:09.312243 26142 raft_consensus.cc:2229] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:18:09.313913 26142 raft_consensus.cc:2258] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:09.314116 26143 tablet_replica.cc:330] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: stopping tablet replica
I20240503 08:18:09.314200 26143 raft_consensus.cc:2229] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:18:09.314291 26143 raft_consensus.cc:2258] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:09.314348 26142 ts_tablet_manager.cc:1875] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:09.315327 26143 ts_tablet_manager.cc:1875] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:09.316256 26142 ts_tablet_manager.cc:1888] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:18:09.316326 26142 log.cc:1196] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/42ec717dbb704dac8d48ea9998723d61
I20240503 08:18:09.316644 26142 ts_tablet_manager.cc:1909] T 42ec717dbb704dac8d48ea9998723d61 P 4c5f84145e4f421493cd48f7a4208615: Deleting consensus metadata
I20240503 08:18:09.316778 26143 ts_tablet_manager.cc:1888] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:18:09.316888 26143 log.cc:1196] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/42ec717dbb704dac8d48ea9998723d61
I20240503 08:18:09.317194 26143 ts_tablet_manager.cc:1909] T 42ec717dbb704dac8d48ea9998723d61 P e1865be0728d4c4bb81748bd63d9a8f4: Deleting consensus metadata
I20240503 08:18:09.317178 25632 catalog_manager.cc:4830] TS 4c5f84145e4f421493cd48f7a4208615 (127.6.68.195:44749): tablet 42ec717dbb704dac8d48ea9998723d61 (table test-table [id=d11702a6c5bd4f9dbb893cc7d40624b6]) successfully deleted
I20240503 08:18:09.317552 25629 catalog_manager.cc:4830] TS e1865be0728d4c4bb81748bd63d9a8f4 (127.6.68.194:33905): tablet 42ec717dbb704dac8d48ea9998723d61 (table test-table [id=d11702a6c5bd4f9dbb893cc7d40624b6]) successfully deleted
W20240503 08:18:09.318295 25630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:39901 (error 108)
I20240503 08:18:09.318776 26114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.318646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:39901 (local address 127.0.0.1:46836)
0503 08:18:09.318700 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.318734 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:39901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20240503 08:18:09.318827 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 25841
W20240503 08:18:09.318904 25630 catalog_manager.cc:4573] TS a97eb980747c441193a804363e3fe1db (127.6.68.193:39901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42ec717dbb704dac8d48ea9998723d61: Network error: Client connection negotiation failed: client connection to 127.6.68.193:39901: connect: Connection refused (error 111)
I20240503 08:18:09.323684 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 25975
I20240503 08:18:09.329136 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 25613
2024-05-03T08:18:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1935 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2024-05-03T08:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:09Z Disabled control of system clock
I20240503 08:18:09.355907 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:41567
--webserver_interface=127.6.68.254
--webserver_port=0
--builtin_ntp_servers=127.6.68.212:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:41567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240503 08:18:09.431612 26148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:09.431834 26148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:09.431866 26148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:09.433140 26148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:09.433213 26148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:09.433233 26148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:09.433251 26148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:09.433271 26148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:09.434672 26148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:41567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:41567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:09.434888 26148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:09.435114 26148 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:09.439659 26153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:09.440039 26156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:09.440855 26154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:09.441380 26148 server_base.cc:1034] running on GCE node
I20240503 08:18:09.441565 26148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:09.441818 26148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:09.442960 26148 hybrid_clock.cc:648] HybridClock initialized: now 1714724289442948 us; error 28 us; skew 500 ppm
I20240503 08:18:09.444154 26148 webserver.cc:460] Webserver started at http://127.6.68.254:33527/ using document root <none> and password file <none>
I20240503 08:18:09.444342 26148 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:09.444412 26148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:09.444538 26148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:09.445587 26148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "49a24437b9b84db9949af0f1c2b152a0"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "5a19fc45513d5dc7a95e70d824b5761a"
server_key_iv: "b74f9980618675b769600fea5045b2b9"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.445909 26148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "49a24437b9b84db9949af0f1c2b152a0"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "5a19fc45513d5dc7a95e70d824b5761a"
server_key_iv: "b74f9980618675b769600fea5045b2b9"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.447176 26148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:09.447937 26162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.448091 26148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:09.448164 26148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "49a24437b9b84db9949af0f1c2b152a0"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "5a19fc45513d5dc7a95e70d824b5761a"
server_key_iv: "b74f9980618675b769600fea5045b2b9"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.448233 26148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:09.460136 26148 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:09.460420 26148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:09.460559 26148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:09.464537 26148 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:41567
I20240503 08:18:09.464565 26214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:41567 every 8 connection(s)
I20240503 08:18:09.464871 26148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:09.465452 26215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:09.467864 26215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0: Bootstrap starting.
I20240503 08:18:09.468374 26215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:09.468624 26215 log.cc:824] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:09.469271 26215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0: No bootstrap required, opened a new log
I20240503 08:18:09.469413 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 26148
I20240503 08:18:09.469529 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:09.469730 6419 external_mini_cluster.cc:1428] Setting key 7033d66f7b1777ed83745af20e9f5c30
I20240503 08:18:09.471114 26215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a24437b9b84db9949af0f1c2b152a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41567 } }
I20240503 08:18:09.471235 26215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:09.471261 26215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a24437b9b84db9949af0f1c2b152a0, State: Initialized, Role: FOLLOWER
I20240503 08:18:09.471361 26215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a24437b9b84db9949af0f1c2b152a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41567 } }
I20240503 08:18:09.471434 26215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:09.471469 26215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:09.471510 26215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:09.472106 26215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a24437b9b84db9949af0f1c2b152a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41567 } }
I20240503 08:18:09.472193 26215 leader_election.cc:302] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49a24437b9b84db9949af0f1c2b152a0; no voters:
I20240503 08:18:09.472329 26215 leader_election.cc:288] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:09.472486 26220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:09.472625 26220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [term 1 LEADER]: Becoming Leader. State: Replica: 49a24437b9b84db9949af0f1c2b152a0, State: Running, Role: LEADER
I20240503 08:18:09.472728 26220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a24437b9b84db9949af0f1c2b152a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41567 } }
I20240503 08:18:09.472615 26215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:09.473095 26221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49a24437b9b84db9949af0f1c2b152a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a24437b9b84db9949af0f1c2b152a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41567 } } }
I20240503 08:18:09.473294 26221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:09.473240 26222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49a24437b9b84db9949af0f1c2b152a0. Latest consensus state: current_term: 1 leader_uuid: "49a24437b9b84db9949af0f1c2b152a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a24437b9b84db9949af0f1c2b152a0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 41567 } } }
I20240503 08:18:09.473541 26222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:09.473567 26224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:09.474046 26224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:09.475735 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.470253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.6.68.254:41567)
0503 08:18:09.470440 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.470446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.470462 (+ 16us) server_negotiation.cc:407] Connection header received
0503 08:18:09.471087 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.471092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.471135 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.471251 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:09.471731 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.473391 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.474456 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.474723 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.474961 (+ 238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:09.474969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:09.474969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:09.475167 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:09.475194 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:09.475201 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:09.475266 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.475487 (+ 221us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.475550 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20240503 08:18:09.475876 26224 catalog_manager.cc:1347] Generated new cluster ID: bf454eae16ef420881c2e322a5320183
I20240503 08:18:09.475917 26224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:09.500088 26224 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:09.500703 26224 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:09.506446 26224 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0: Generated new TSK 0
I20240503 08:18:09.506644 26224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:09.514565 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.193:0
--local_ip_for_outbound_sockets=127.6.68.193
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:41567
--builtin_ntp_servers=127.6.68.212:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240503 08:18:09.593564 26239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:09.593747 26239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:09.593782 26239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:09.595038 26239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:09.595081 26239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:09.595119 26239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.193
I20240503 08:18:09.596562 26239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.68.193
--webserver_port=0
--tserver_master_addrs=127.6.68.254:41567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.193
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:09.596789 26239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:09.597041 26239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:09.602103 26245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:09.602499 26247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:09.603147 26244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:09.603709 26239 server_base.cc:1034] running on GCE node
I20240503 08:18:09.603881 26239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:09.604108 26239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:09.605257 26239 hybrid_clock.cc:648] HybridClock initialized: now 1714724289605235 us; error 41 us; skew 500 ppm
I20240503 08:18:09.606627 26239 webserver.cc:460] Webserver started at http://127.6.68.193:35085/ using document root <none> and password file <none>
I20240503 08:18:09.606813 26239 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:09.606859 26239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:09.606966 26239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:09.608031 26239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/data/instance:
uuid: "b3f9ea0829ee4b938b77c24ceb2428b2"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "f4b2bf1faf79b47ac4f6d1d3619d8322"
server_key_iv: "7c0d2f2e97772a4ae3901fa40fddd7e1"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.608314 26239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance:
uuid: "b3f9ea0829ee4b938b77c24ceb2428b2"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "f4b2bf1faf79b47ac4f6d1d3619d8322"
server_key_iv: "7c0d2f2e97772a4ae3901fa40fddd7e1"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.609427 26239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:09.610085 26253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.610291 26239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:09.610386 26239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/wal
uuid: "b3f9ea0829ee4b938b77c24ceb2428b2"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "f4b2bf1faf79b47ac4f6d1d3619d8322"
server_key_iv: "7c0d2f2e97772a4ae3901fa40fddd7e1"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.610455 26239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:09.624238 26239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:09.624503 26239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:09.624594 26239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:09.624791 26239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:09.625141 26239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:09.625176 26239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.625198 26239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:09.625214 26239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.632016 26239 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.193:42095
I20240503 08:18:09.632087 26366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.193:42095 every 8 connection(s)
I20240503 08:18:09.632356 26239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/data/info.pb
I20240503 08:18:09.636806 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.633011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:47041 (local address 127.6.68.254:41567)
0503 08:18:09.633058 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.633061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.633272 (+ 211us) server_negotiation.cc:407] Connection header received
0503 08:18:09.633760 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.633761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.633775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.633834 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:09.634300 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.634781 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.635315 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.635496 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.636533 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:09.636537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:09.636538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:09.636542 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:09.636556 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:09.636560 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:09.636613 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.636694 (+ 81us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.636731 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240503 08:18:09.636955 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.632960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:41567 (local address 127.6.68.193:47041)
0503 08:18:09.633169 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.633199 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.633655 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.633905 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.633908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.633916 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:09.634210 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.634212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.634873 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.634874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.635228 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.635229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.635358 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.636394 (+ 1036us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:18:09.636395 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:09.636419 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:18:09.636441 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:09.636618 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:09.636622 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:09.636625 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.636694 (+ 69us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.636768 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20240503 08:18:09.637470 26367 heartbeater.cc:347] Connected to a master server at 127.6.68.254:41567
I20240503 08:18:09.637547 26367 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:09.637728 26367 heartbeater.cc:511] Master 127.6.68.254:41567 requested a full tablet report, sending...
I20240503 08:18:09.638177 26179 ts_manager.cc:194] Registered new tserver with Master: b3f9ea0829ee4b938b77c24ceb2428b2 (127.6.68.193:42095)
I20240503 08:18:09.638790 26179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.193:47041
I20240503 08:18:09.639107 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 26239
I20240503 08:18:09.639202 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-0/wal/instance
I20240503 08:18:09.639362 6419 external_mini_cluster.cc:1428] Setting key de98953585539e50eedcfbf94bb7a908
I20240503 08:18:09.640203 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.194:0
--local_ip_for_outbound_sockets=127.6.68.194
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:41567
--builtin_ntp_servers=127.6.68.212:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240503 08:18:09.718362 26370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:09.718565 26370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:09.718595 26370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:09.719871 26370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:09.719909 26370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:09.719945 26370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.194
I20240503 08:18:09.721350 26370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.68.194
--webserver_port=0
--tserver_master_addrs=127.6.68.254:41567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.194
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:09.721570 26370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:09.721799 26370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:09.726758 26375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:09.727209 26376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:09.727761 26378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:09.728317 26370 server_base.cc:1034] running on GCE node
I20240503 08:18:09.728508 26370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:09.728714 26370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:09.729843 26370 hybrid_clock.cc:648] HybridClock initialized: now 1714724289729823 us; error 37 us; skew 500 ppm
I20240503 08:18:09.731117 26370 webserver.cc:460] Webserver started at http://127.6.68.194:43427/ using document root <none> and password file <none>
I20240503 08:18:09.731312 26370 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:09.731359 26370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:09.731468 26370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:09.732513 26370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/data/instance:
uuid: "bd6cb6556ce44311bee4fb72e853438e"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "c8425647f4fd332b16dd9dba3fec59ae"
server_key_iv: "af36f36544ae3c8ea0459052841459ca"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.732825 26370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance:
uuid: "bd6cb6556ce44311bee4fb72e853438e"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "c8425647f4fd332b16dd9dba3fec59ae"
server_key_iv: "af36f36544ae3c8ea0459052841459ca"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.734012 26370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:09.734683 26384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.734849 26370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:09.734920 26370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/wal
uuid: "bd6cb6556ce44311bee4fb72e853438e"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "c8425647f4fd332b16dd9dba3fec59ae"
server_key_iv: "af36f36544ae3c8ea0459052841459ca"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.734980 26370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:09.745469 26370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:09.745711 26370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:09.745826 26370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:09.746014 26370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:09.746307 26370 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:09.746347 26370 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.746381 26370 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:09.746413 26370 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.751945 26370 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.194:34601
I20240503 08:18:09.751989 26497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.194:34601 every 8 connection(s)
I20240503 08:18:09.752290 26370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/data/info.pb
I20240503 08:18:09.754441 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 26370
I20240503 08:18:09.754544 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/wal/instance
I20240503 08:18:09.754719 6419 external_mini_cluster.cc:1428] Setting key e2687c6dded719013cf7b79015c67384
I20240503 08:18:09.755710 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.68.195:0
--local_ip_for_outbound_sockets=127.6.68.195
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:41567
--builtin_ntp_servers=127.6.68.212:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240503 08:18:09.756863 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.752963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.194:50193 (local address 127.6.68.254:41567)
0503 08:18:09.753021 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.753023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.753196 (+ 173us) server_negotiation.cc:407] Connection header received
0503 08:18:09.753616 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.753617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.753625 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.753694 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:09.754132 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.754872 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.755466 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.756077 (+ 611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.756552 (+ 475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:09.756556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:09.756557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:09.756562 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:09.756575 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:09.756579 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:09.756643 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.756709 (+ 66us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.756746 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240503 08:18:09.756871 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.752927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:41567 (local address 127.6.68.194:50193)
0503 08:18:09.753099 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.753126 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.753538 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.753749 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.753753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.753760 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:09.754064 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.754067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.754962 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.754963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.755390 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.755392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.755500 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.756433 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:18:09.756433 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:09.756451 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:18:09.756468 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:09.756644 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:09.756645 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:09.756647 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.756697 (+ 50us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.756755 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20240503 08:18:09.757687 26498 heartbeater.cc:347] Connected to a master server at 127.6.68.254:41567
I20240503 08:18:09.757771 26498 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:09.757959 26498 heartbeater.cc:511] Master 127.6.68.254:41567 requested a full tablet report, sending...
I20240503 08:18:09.758302 26179 ts_manager.cc:194] Registered new tserver with Master: bd6cb6556ce44311bee4fb72e853438e (127.6.68.194:34601)
I20240503 08:18:09.758853 26179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.194:50193
W20240503 08:18:09.832660 26501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:09.832819 26501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:09.832859 26501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:09.834048 26501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:09.834082 26501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:09.834105 26501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.68.195
I20240503 08:18:09.835441 26501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.68.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.68.195
--webserver_port=0
--tserver_master_addrs=127.6.68.254:41567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.68.195
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:09.835602 26501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:09.835832 26501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:09.840579 26509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:09.841039 26506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:09.841665 26507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:09.842314 26501 server_base.cc:1034] running on GCE node
I20240503 08:18:09.842512 26501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:09.842691 26501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:09.843820 26501 hybrid_clock.cc:648] HybridClock initialized: now 1714724289843804 us; error 36 us; skew 500 ppm
I20240503 08:18:09.845237 26501 webserver.cc:460] Webserver started at http://127.6.68.195:41147/ using document root <none> and password file <none>
I20240503 08:18:09.845440 26501 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:09.845484 26501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:09.845602 26501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:09.846580 26501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/data/instance:
uuid: "f8b7fb93b8f0431a80ea27241a7e9dab"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "041a7d96c31fb25d176d759d56965b5f"
server_key_iv: "e2c18c3c5dc8d11b72fe8610038acba8"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.846875 26501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance:
uuid: "f8b7fb93b8f0431a80ea27241a7e9dab"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "041a7d96c31fb25d176d759d56965b5f"
server_key_iv: "e2c18c3c5dc8d11b72fe8610038acba8"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.848094 26501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:18:09.848788 26515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.848953 26501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:09.849018 26501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/wal
uuid: "f8b7fb93b8f0431a80ea27241a7e9dab"
format_stamp: "Formatted at 2024-05-03 08:18:09 on dist-test-slave-3g67"
server_key: "041a7d96c31fb25d176d759d56965b5f"
server_key_iv: "e2c18c3c5dc8d11b72fe8610038acba8"
server_key_version: "encryptionkey@0"
I20240503 08:18:09.849090 26501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:09.876755 26501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:09.877022 26501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:09.877149 26501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:09.877352 26501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:18:09.877671 26501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:18:09.877703 26501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.877738 26501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:18:09.877764 26501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:09.882988 26501 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.195:37279
I20240503 08:18:09.883066 26628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.195:37279 every 8 connection(s)
I20240503 08:18:09.883352 26501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/data/info.pb
I20240503 08:18:09.888124 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.884112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.195:60279 (local address 127.6.68.254:41567)
0503 08:18:09.884198 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.884201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.884408 (+ 207us) server_negotiation.cc:407] Connection header received
0503 08:18:09.884889 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.884890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.884905 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.884983 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:09.885481 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.885974 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.886528 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.886702 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.887763 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:09.887768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:09.887768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:09.887773 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:09.887793 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:09.887797 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:09.887859 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.887991 (+ 132us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.888030 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240503 08:18:09.888255 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.884057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:41567 (local address 127.6.68.195:60279)
0503 08:18:09.884279 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.884304 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.884780 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.885037 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.885042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.885054 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:09.885374 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.885378 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.886069 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.886070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.886440 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.886442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.886581 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.887547 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0503 08:18:09.887548 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:09.887568 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0503 08:18:09.887587 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:09.887874 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:09.887880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:09.887882 (+ 2us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.887990 (+ 108us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.888070 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20240503 08:18:09.888772 26629 heartbeater.cc:347] Connected to a master server at 127.6.68.254:41567
I20240503 08:18:09.888857 26629 heartbeater.cc:464] Registering TS with master...
I20240503 08:18:09.889065 26629 heartbeater.cc:511] Master 127.6.68.254:41567 requested a full tablet report, sending...
I20240503 08:18:09.889449 26179 ts_manager.cc:194] Registered new tserver with Master: f8b7fb93b8f0431a80ea27241a7e9dab (127.6.68.195:37279)
I20240503 08:18:09.889853 26179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.68.195:60279
I20240503 08:18:09.890429 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 26501
I20240503 08:18:09.890522 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/wal/instance
I20240503 08:18:09.890681 6419 external_mini_cluster.cc:1428] Setting key 2e3057bce93598773d475fb77cbc7175
I20240503 08:18:09.891525 6419 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 08:18:09.895167 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.892770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.6.68.254:41567)
0503 08:18:09.892834 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.892836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.892906 (+ 70us) server_negotiation.cc:407] Connection header received
0503 08:18:09.892978 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.892979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.892989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.893041 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:09.893360 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.893833 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.894363 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.894553 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.894931 (+ 378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:09.894934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:09.894935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:09.894938 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:09.894950 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:09.894956 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:09.895003 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.895063 (+ 60us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.895086 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240503 08:18:09.896677 26179 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:18:09.898146 26179 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:18:09.903308 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.900334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.6.68.193:42095)
0503 08:18:09.900497 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.900505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.900517 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:18:09.900548 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.900549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.900561 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.900630 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:09.901000 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.901836 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.902561 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.903063 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.903161 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.903192 (+ 31us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.903231 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20240503 08:18:09.903342 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.899998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:37279 (local address 127.0.0.1:56576)
0503 08:18:09.900169 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.900194 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.900261 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.900691 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.900692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.900695 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:09.900890 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.900891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.902189 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.902190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.903129 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.903131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.903208 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.903211 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.903241 (+ 30us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.903276 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20240503 08:18:09.902766 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.900238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:42095 (local address 127.0.0.1:45408)
0503 08:18:09.900372 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.900381 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.900426 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.900660 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.900663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.900665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:09.900910 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.900912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.901913 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.901914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.902477 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.902478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.902547 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.902592 (+ 45us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.902647 (+ 55us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.902689 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20240503 08:18:09.903914 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.900049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:34601 (local address 127.0.0.1:36066)
0503 08:18:09.900371 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:09.900381 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:09.900426 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:09.901044 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:09.901046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:09.901048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:09.901256 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.901258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.902574 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.902575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:09.903688 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:09.903691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.903769 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.903772 (+ 3us) client_negotiation.cc:769] Sending connection context
0503 08:18:09.903803 (+ 31us) client_negotiation.cc:240] Negotiation successful
0503 08:18:09.903841 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":89,"threads_started":1}
I20240503 08:18:09.903971 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.900091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.6.68.195:37279)
0503 08:18:09.900533 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.900538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.900545 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:09.900574 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.900575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.900597 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.900676 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:09.900978 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.902094 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.903217 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.903753 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.903827 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.903854 (+ 27us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.903887 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":80,"threads_started":1}
I20240503 08:18:09.904247 26301 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe96ee5c7da45cd8b69d024910016ee (DEFAULT_TABLE table=test-table [id=7c79e77599844333b3bb0fce2c678ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:09.904384 26563 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe96ee5c7da45cd8b69d024910016ee (DEFAULT_TABLE table=test-table [id=7c79e77599844333b3bb0fce2c678ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:09.904527 26301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe96ee5c7da45cd8b69d024910016ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:09.904639 26563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe96ee5c7da45cd8b69d024910016ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:09.906839 26644 tablet_bootstrap.cc:492] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2: Bootstrap starting.
I20240503 08:18:09.907358 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:09.900162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.6.68.194:34601)
0503 08:18:09.900813 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:09.900819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:09.900828 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:09.900862 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:09.900863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:09.900878 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:09.900955 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:09.901342 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.902490 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:09.904019 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:09.907168 (+ 3149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:09.907237 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:09.907269 (+ 32us) server_negotiation.cc:299] Negotiation successful
0503 08:18:09.907300 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":90,"threads_started":1}
I20240503 08:18:09.907491 26644 tablet_bootstrap.cc:654] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:09.907794 26643 tablet_bootstrap.cc:492] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab: Bootstrap starting.
I20240503 08:18:09.907807 26644 log.cc:824] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:09.908002 26432 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe96ee5c7da45cd8b69d024910016ee (DEFAULT_TABLE table=test-table [id=7c79e77599844333b3bb0fce2c678ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:18:09.908273 26432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe96ee5c7da45cd8b69d024910016ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:09.908332 26643 tablet_bootstrap.cc:654] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:09.908387 26644 tablet_bootstrap.cc:492] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2: No bootstrap required, opened a new log
I20240503 08:18:09.908428 26644 ts_tablet_manager.cc:1367] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:18:09.908589 26643 log.cc:824] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:09.909302 26643 tablet_bootstrap.cc:492] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab: No bootstrap required, opened a new log
I20240503 08:18:09.909356 26643 ts_tablet_manager.cc:1367] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:18:09.910207 26644 raft_consensus.cc:348] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 37279 } } peers { permanent_uuid: "bd6cb6556ce44311bee4fb72e853438e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34601 } } peers { permanent_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42095 } }
I20240503 08:18:09.910308 26644 raft_consensus.cc:374] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:09.910323 26644 raft_consensus.cc:729] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f9ea0829ee4b938b77c24ceb2428b2, State: Initialized, Role: FOLLOWER
I20240503 08:18:09.910403 26644 consensus_queue.cc:260] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 37279 } } peers { permanent_uuid: "bd6cb6556ce44311bee4fb72e853438e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34601 } } peers { permanent_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42095 } }
I20240503 08:18:09.910617 26644 ts_tablet_manager.cc:1398] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:18:09.911190 26648 tablet_bootstrap.cc:492] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: Bootstrap starting.
I20240503 08:18:09.911240 26367 heartbeater.cc:503] Master 127.6.68.254:41567 was elected leader, sending a full tablet report...
I20240503 08:18:09.911764 26648 tablet_bootstrap.cc:654] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:09.911680 26643 raft_consensus.cc:348] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 37279 } } peers { permanent_uuid: "bd6cb6556ce44311bee4fb72e853438e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34601 } } peers { permanent_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42095 } }
I20240503 08:18:09.911813 26643 raft_consensus.cc:374] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:09.911840 26643 raft_consensus.cc:729] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b7fb93b8f0431a80ea27241a7e9dab, State: Initialized, Role: FOLLOWER
I20240503 08:18:09.911967 26643 consensus_queue.cc:260] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 37279 } } peers { permanent_uuid: "bd6cb6556ce44311bee4fb72e853438e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34601 } } peers { permanent_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42095 } }
I20240503 08:18:09.912045 26648 log.cc:824] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:09.912189 26643 ts_tablet_manager.cc:1398] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:18:09.912271 26629 heartbeater.cc:503] Master 127.6.68.254:41567 was elected leader, sending a full tablet report...
I20240503 08:18:09.912777 26648 tablet_bootstrap.cc:492] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: No bootstrap required, opened a new log
I20240503 08:18:09.912843 26648 ts_tablet_manager.cc:1367] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:18:09.914662 26648 raft_consensus.cc:348] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 37279 } } peers { permanent_uuid: "bd6cb6556ce44311bee4fb72e853438e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34601 } } peers { permanent_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42095 } }
I20240503 08:18:09.914767 26648 raft_consensus.cc:374] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:09.914783 26648 raft_consensus.cc:729] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6cb6556ce44311bee4fb72e853438e, State: Initialized, Role: FOLLOWER
I20240503 08:18:09.914851 26648 consensus_queue.cc:260] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 37279 } } peers { permanent_uuid: "bd6cb6556ce44311bee4fb72e853438e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34601 } } peers { permanent_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42095 } }
I20240503 08:18:09.915060 26648 ts_tablet_manager.cc:1398] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 08:18:09.915145 26498 heartbeater.cc:503] Master 127.6.68.254:41567 was elected leader, sending a full tablet report...
W20240503 08:18:10.002823 26499 tablet.cc:2369] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:18:10.066313 26647 raft_consensus.cc:482] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:18:10.066449 26647 raft_consensus.cc:504] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 37279 } } peers { permanent_uuid: "bd6cb6556ce44311bee4fb72e853438e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34601 } } peers { permanent_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42095 } }
I20240503 08:18:10.066766 26647 leader_election.cc:288] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8b7fb93b8f0431a80ea27241a7e9dab (127.6.68.195:37279), bd6cb6556ce44311bee4fb72e853438e (127.6.68.194:34601)
I20240503 08:18:10.069132 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:10.066990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.195:37279 (local address 127.6.68.193:40529)
0503 08:18:10.067051 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:10.067065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:10.067134 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:10.067286 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:10.067287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:10.067289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:10.067520 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:10.067522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:10.068268 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:10.068270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:10.068917 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:10.068919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:10.068980 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:10.068985 (+ 5us) client_negotiation.cc:769] Sending connection context
0503 08:18:10.069017 (+ 32us) client_negotiation.cc:240] Negotiation successful
0503 08:18:10.069052 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20240503 08:18:10.069267 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:10.066994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.194:34601 (local address 127.6.68.193:59579)
0503 08:18:10.067180 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:10.067189 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:10.067257 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:10.067397 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:10.067397 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:10.067399 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 08:18:10.067609 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:10.067611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:10.068358 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:10.068359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:10.068939 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:10.068940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:10.069036 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:10.069040 (+ 4us) client_negotiation.cc:769] Sending connection context
0503 08:18:10.069074 (+ 34us) client_negotiation.cc:240] Negotiation successful
0503 08:18:10.069108 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":13,"thread_start_us":105,"threads_started":1}
I20240503 08:18:10.069618 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:10.067142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:59579 (local address 127.6.68.194:34601)
0503 08:18:10.067183 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:10.067185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:10.067265 (+ 80us) server_negotiation.cc:407] Connection header received
0503 08:18:10.067312 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:10.067313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:10.067322 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:10.067385 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:10.067723 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:10.068255 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:10.069047 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:10.069523 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:10.069530 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:10.069549 (+ 19us) server_negotiation.cc:299] Negotiation successful
0503 08:18:10.069571 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:18:10.069633 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:10.067102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.68.193:40529 (local address 127.6.68.195:37279)
0503 08:18:10.067152 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:10.067155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:10.067166 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:18:10.067201 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:10.067202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:10.067216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:10.067276 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0503 08:18:10.067611 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:10.068167 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:10.069017 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:10.069502 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:10.069509 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:10.069532 (+ 23us) server_negotiation.cc:299] Negotiation successful
0503 08:18:10.069562 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240503 08:18:10.069914 26583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe96ee5c7da45cd8b69d024910016ee" candidate_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" is_pre_election: true
I20240503 08:18:10.069913 26452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe96ee5c7da45cd8b69d024910016ee" candidate_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6cb6556ce44311bee4fb72e853438e" is_pre_election: true
I20240503 08:18:10.070046 26452 raft_consensus.cc:2454] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f9ea0829ee4b938b77c24ceb2428b2 in term 0.
I20240503 08:18:10.070050 26583 raft_consensus.cc:2454] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f9ea0829ee4b938b77c24ceb2428b2 in term 0.
I20240503 08:18:10.070290 26255 leader_election.cc:302] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3f9ea0829ee4b938b77c24ceb2428b2, bd6cb6556ce44311bee4fb72e853438e; no voters:
I20240503 08:18:10.070477 26647 raft_consensus.cc:2781] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:18:10.070571 26647 raft_consensus.cc:482] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:18:10.070598 26647 raft_consensus.cc:3037] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:10.071254 26647 raft_consensus.cc:504] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 37279 } } peers { permanent_uuid: "bd6cb6556ce44311bee4fb72e853438e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34601 } } peers { permanent_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42095 } }
I20240503 08:18:10.071405 26647 leader_election.cc:288] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [CANDIDATE]: Term 1 election: Requested vote from peers f8b7fb93b8f0431a80ea27241a7e9dab (127.6.68.195:37279), bd6cb6556ce44311bee4fb72e853438e (127.6.68.194:34601)
I20240503 08:18:10.071583 26452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe96ee5c7da45cd8b69d024910016ee" candidate_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6cb6556ce44311bee4fb72e853438e"
I20240503 08:18:10.071579 26583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe96ee5c7da45cd8b69d024910016ee" candidate_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab"
I20240503 08:18:10.071672 26452 raft_consensus.cc:3037] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:10.071676 26583 raft_consensus.cc:3037] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:10.072391 26583 raft_consensus.cc:2454] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f9ea0829ee4b938b77c24ceb2428b2 in term 1.
I20240503 08:18:10.072455 26452 raft_consensus.cc:2454] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f9ea0829ee4b938b77c24ceb2428b2 in term 1.
I20240503 08:18:10.072628 26255 leader_election.cc:302] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3f9ea0829ee4b938b77c24ceb2428b2, bd6cb6556ce44311bee4fb72e853438e; no voters:
I20240503 08:18:10.072727 26647 raft_consensus.cc:2781] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:10.072877 26647 raft_consensus.cc:686] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 1 LEADER]: Becoming Leader. State: Replica: b3f9ea0829ee4b938b77c24ceb2428b2, State: Running, Role: LEADER
I20240503 08:18:10.072968 26647 consensus_queue.cc:237] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 37279 } } peers { permanent_uuid: "bd6cb6556ce44311bee4fb72e853438e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34601 } } peers { permanent_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42095 } }
I20240503 08:18:10.073809 26179 catalog_manager.cc:5484] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3f9ea0829ee4b938b77c24ceb2428b2 (127.6.68.193). New cstate: current_term: 1 leader_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd6cb6556ce44311bee4fb72e853438e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3f9ea0829ee4b938b77c24ceb2428b2" member_type: VOTER last_known_addr { host: "127.6.68.193" port: 42095 } health_report { overall_health: HEALTHY } } }
I20240503 08:18:10.087203 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:10.085086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.6.68.193:42095)
0503 08:18:10.085158 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:10.085160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:10.085166 (+ 6us) server_negotiation.cc:407] Connection header received
0503 08:18:10.085205 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:10.085206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:10.085223 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:10.085303 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:10.085598 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:10.086097 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:10.086685 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:10.086825 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:10.086863 (+ 38us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:10.086952 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:10.087025 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:10.087098 (+ 73us) server_negotiation.cc:299] Negotiation successful
0503 08:18:10.087122 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240503 08:18:10.089006 26583 raft_consensus.cc:1261] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab [term 1 FOLLOWER]: Refusing update from remote peer b3f9ea0829ee4b938b77c24ceb2428b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:10.089010 26452 raft_consensus.cc:1261] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e [term 1 FOLLOWER]: Refusing update from remote peer b3f9ea0829ee4b938b77c24ceb2428b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:10.089335 26647 consensus_queue.cc:1035] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6cb6556ce44311bee4fb72e853438e" member_type: VOTER last_known_addr { host: "127.6.68.194" port: 34601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:10.089463 26653 consensus_queue.cc:1035] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8b7fb93b8f0431a80ea27241a7e9dab" member_type: VOTER last_known_addr { host: "127.6.68.195" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:10.091526 26656 mvcc.cc:205] Tried to move back new op lower bound from 7023510692201881600 to 7023510692139388928. Current Snapshot: MvccSnapshot[applied={T|T < 7023510692201881600}]
I20240503 08:18:10.091936 26658 mvcc.cc:205] Tried to move back new op lower bound from 7023510692201881600 to 7023510692139388928. Current Snapshot: MvccSnapshot[applied={T|T < 7023510692201881600}]
I20240503 08:18:10.092826 26657 mvcc.cc:205] Tried to move back new op lower bound from 7023510692201881600 to 7023510692139388928. Current Snapshot: MvccSnapshot[applied={T|T < 7023510692201881600}]
I20240503 08:18:10.093683 26177 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60604:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:10.093775 26177 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60604:
table { table_name: "test-table" } modify_external_catalogs: true
I20240503 08:18:10.094506 26177 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 49a24437b9b84db9949af0f1c2b152a0: Sending DeleteTablet for 3 replicas of tablet 6fe96ee5c7da45cd8b69d024910016ee
I20240503 08:18:10.094802 26563 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe96ee5c7da45cd8b69d024910016ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:10 UTC) from {username='slave'} at 127.0.0.1:56576
I20240503 08:18:10.094853 26301 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe96ee5c7da45cd8b69d024910016ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:10 UTC) from {username='slave'} at 127.0.0.1:45408
I20240503 08:18:10.094858 26432 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe96ee5c7da45cd8b69d024910016ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:18:10 UTC) from {username='slave'} at 127.0.0.1:36066
I20240503 08:18:10.095127 26667 tablet_replica.cc:330] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2: stopping tablet replica
I20240503 08:18:10.095127 26666 tablet_replica.cc:330] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab: stopping tablet replica
I20240503 08:18:10.095293 26666 raft_consensus.cc:2229] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:18:10.095245 26667 raft_consensus.cc:2229] T 6fe96ee5c7da45cd8b69d024910016ee P b3f9ea0829ee4b938b77c24ceb2428b2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:18:10.095352 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 26239
I20240503 08:18:10.095388 26666 raft_consensus.cc:2258] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:10.096192 26668 tablet_replica.cc:330] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: stopping tablet replica
I20240503 08:18:10.096287 26668 raft_consensus.cc:2229] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:18:10.096377 26668 raft_consensus.cc:2258] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:18:10.096699 26666 ts_tablet_manager.cc:1875] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:10.096788 26668 ts_tablet_manager.cc:1875] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:18:10.098475 26666 ts_tablet_manager.cc:1888] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:18:10.098491 26668 ts_tablet_manager.cc:1888] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240503 08:18:10.098547 26668 log.cc:1196] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-1/wal/wals/6fe96ee5c7da45cd8b69d024910016ee
I20240503 08:18:10.098600 26666 log.cc:1196] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab: Deleting WAL directory at /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714724013808011-6419-0/minicluster-data/ts-2/wal/wals/6fe96ee5c7da45cd8b69d024910016ee
I20240503 08:18:10.098845 26668 ts_tablet_manager.cc:1909] T 6fe96ee5c7da45cd8b69d024910016ee P bd6cb6556ce44311bee4fb72e853438e: Deleting consensus metadata
I20240503 08:18:10.098875 26666 ts_tablet_manager.cc:1909] T 6fe96ee5c7da45cd8b69d024910016ee P f8b7fb93b8f0431a80ea27241a7e9dab: Deleting consensus metadata
I20240503 08:18:10.099227 26164 catalog_manager.cc:4830] TS bd6cb6556ce44311bee4fb72e853438e (127.6.68.194:34601): tablet 6fe96ee5c7da45cd8b69d024910016ee (table test-table [id=7c79e77599844333b3bb0fce2c678ef3]) successfully deleted
I20240503 08:18:10.099364 26166 catalog_manager.cc:4830] TS f8b7fb93b8f0431a80ea27241a7e9dab (127.6.68.195:37279): tablet 6fe96ee5c7da45cd8b69d024910016ee (table test-table [id=7c79e77599844333b3bb0fce2c678ef3]) successfully deleted
W20240503 08:18:10.101948 26164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.68.193:42095 (error 108)
I20240503 08:18:10.102447 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:10.102319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.193:42095 (local address 127.0.0.1:45426)
0503 08:18:10.102376 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:10.102417 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.193:42095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20240503 08:18:10.102456 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 26370
W20240503 08:18:10.103608 26164 catalog_manager.cc:4573] TS b3f9ea0829ee4b938b77c24ceb2428b2 (127.6.68.193:42095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6fe96ee5c7da45cd8b69d024910016ee: Network error: Client connection negotiation failed: client connection to 127.6.68.193:42095: connect: Connection refused (error 111)
I20240503 08:18:10.107497 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 26501
I20240503 08:18:10.112066 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 26148
2024-05-03T08:18:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (782 ms)
[----------] 6 tests from EncryptionPolicyTest (8155 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:10 dist-test-slave-3g67 krb5kdc[26672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:10 dist-test-slave-3g67 krb5kdc[26672](info): set up 2 sockets
May 03 08:18:10 dist-test-slave-3g67 krb5kdc[26672](info): commencing operation
krb5kdc: starting...
W20240503 08:18:11.083982 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:11 dist-test-slave-3g67 krb5kdc[26672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:11.203269 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:11.280953 26685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:11.281172 26685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:11.281194 26685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:11.282485 26685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:11.282531 26685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:11.282543 26685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:11.282555 26685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:11.282567 26685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:11.284159 26685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:11.284353 26685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:11.284572 26685 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:11.289115 26693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:11.289856 26691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:11.290483 26690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:11.291163 26685 server_base.cc:1034] running on GCE node
I20240503 08:18:11.291378 26685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:11.291602 26685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:11.292766 26685 hybrid_clock.cc:648] HybridClock initialized: now 1714724291292756 us; error 30 us; skew 500 ppm
May 03 08:18:11 dist-test-slave-3g67 krb5kdc[26672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:11.295038 26685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:11.295388 26685 webserver.cc:460] Webserver started at http://127.0.0.1:44431/ using document root <none> and password file <none>
I20240503 08:18:11.295545 26685 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:11.295586 26685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:11.295702 26685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:11.296624 26685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "a37416aa3b65498aa472343de0007416"
format_stamp: "Formatted at 2024-05-03 08:18:11 on dist-test-slave-3g67"
server_key: "40ec9522a56d61e9c273cd483b2ec191"
server_key_iv: "83f091ad1932b3aa9e00e5025490c848"
server_key_version: "encryptionkey@0"
I20240503 08:18:11.296882 26685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "a37416aa3b65498aa472343de0007416"
format_stamp: "Formatted at 2024-05-03 08:18:11 on dist-test-slave-3g67"
server_key: "40ec9522a56d61e9c273cd483b2ec191"
server_key_iv: "83f091ad1932b3aa9e00e5025490c848"
server_key_version: "encryptionkey@0"
I20240503 08:18:11.298017 26685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:11.298671 26700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:11.298858 26685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:11.298939 26685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "a37416aa3b65498aa472343de0007416"
format_stamp: "Formatted at 2024-05-03 08:18:11 on dist-test-slave-3g67"
server_key: "40ec9522a56d61e9c273cd483b2ec191"
server_key_iv: "83f091ad1932b3aa9e00e5025490c848"
server_key_version: "encryptionkey@0"
I20240503 08:18:11.298995 26685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:11.305970 26685 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:11.308056 26685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:11.308154 26685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:11.311767 26685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35257
I20240503 08:18:11.311792 26752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35257 every 8 connection(s)
I20240503 08:18:11.312108 26685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:11.312776 26753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:11.314692 26753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416: Bootstrap starting.
I20240503 08:18:11.315202 26753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:11.315439 26753 log.cc:824] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:11.316076 26753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416: No bootstrap required, opened a new log
I20240503 08:18:11.317407 26753 raft_consensus.cc:348] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37416aa3b65498aa472343de0007416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } }
I20240503 08:18:11.317535 26753 raft_consensus.cc:374] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:11.317564 26753 raft_consensus.cc:729] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37416aa3b65498aa472343de0007416, State: Initialized, Role: FOLLOWER
I20240503 08:18:11.317613 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 26685
I20240503 08:18:11.317678 26753 consensus_queue.cc:260] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37416aa3b65498aa472343de0007416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } }
I20240503 08:18:11.317734 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:11.317747 26753 raft_consensus.cc:388] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:11.317792 26753 raft_consensus.cc:482] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:11.317844 26753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:11.318006 6419 external_mini_cluster.cc:1428] Setting key 6ac6bf088f474bc3e859e7621104ebbb
I20240503 08:18:11.318517 26753 raft_consensus.cc:504] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37416aa3b65498aa472343de0007416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } }
I20240503 08:18:11.318676 26753 leader_election.cc:302] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a37416aa3b65498aa472343de0007416; no voters:
I20240503 08:18:11.318836 26753 leader_election.cc:288] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:11.319044 26758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:11.319051 26753 sys_catalog.cc:564] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:11.319293 26758 raft_consensus.cc:686] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [term 1 LEADER]: Becoming Leader. State: Replica: a37416aa3b65498aa472343de0007416, State: Running, Role: LEADER
I20240503 08:18:11.319430 26758 consensus_queue.cc:237] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37416aa3b65498aa472343de0007416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } }
I20240503 08:18:11.319813 26759 sys_catalog.cc:455] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a37416aa3b65498aa472343de0007416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37416aa3b65498aa472343de0007416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } } }
I20240503 08:18:11.319828 26760 sys_catalog.cc:455] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a37416aa3b65498aa472343de0007416. Latest consensus state: current_term: 1 leader_uuid: "a37416aa3b65498aa472343de0007416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37416aa3b65498aa472343de0007416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } } }
I20240503 08:18:11.319936 26759 sys_catalog.cc:458] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:11.319945 26760 sys_catalog.cc:458] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:11.320374 26767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:11.320858 26767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:11.322304 26767 catalog_manager.cc:1347] Generated new cluster ID: 94c85348099f4a1ca363c08fd6661ca0
I20240503 08:18:11.322353 26767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 03 08:18:11 dist-test-slave-3g67 krb5kdc[26672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240503 08:18:11.327601 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:11.318448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52353 (local address 127.0.0.1:35257)
0503 08:18:11.318649 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:11.318660 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:11.318704 (+ 44us) server_negotiation.cc:407] Connection header received
0503 08:18:11.318805 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:11.318811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:11.318859 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:11.318998 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:11.320047 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:11.321292 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:11.321948 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:11.322575 (+ 627us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:11.324455 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:11.324486 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:11.324486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:11.324507 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:11.326250 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:11.326618 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:11.326621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:11.326621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:11.326667 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:11.326857 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:11.326858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:11.326858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:11.327058 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:11.327189 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:11.327366 (+ 177us) server_negotiation.cc:299] Negotiation successful
0503 08:18:11.327463 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20240503 08:18:11.328846 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:11.337976 26767 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:11.338696 26767 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:11.345038 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:11.337892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60899 (local address 127.0.0.1:35257)
0503 08:18:11.337935 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:11.337937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:11.338107 (+ 170us) server_negotiation.cc:407] Connection header received
0503 08:18:11.338229 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:11.338230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:11.338241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:11.338310 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:11.339248 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:11.340058 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:11.340765 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:11.340954 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:11.342166 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:11.342169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:11.342169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:11.342174 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:11.343908 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:11.344301 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:11.344302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:11.344302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:11.344323 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:11.344511 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:11.344512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:11.344512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:11.344621 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:11.344726 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:11.344875 (+ 149us) server_negotiation.cc:299] Negotiation successful
0503 08:18:11.344940 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:18:11.348583 26767 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a37416aa3b65498aa472343de0007416: Generated new TSK 0
I20240503 08:18:11.348764 26767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:11.350718 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 26685
W20240503 08:18:11.355206 26670 connection.cc:535] client connection to 127.0.0.1:35257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35257): Connection reset by peer (error 104)
2024-05-03T08:18:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1237 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:11 dist-test-slave-3g67 krb5kdc[26784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:11 dist-test-slave-3g67 krb5kdc[26784](info): set up 2 sockets
May 03 08:18:11 dist-test-slave-3g67 krb5kdc[26784](info): commencing operation
krb5kdc: starting...
W20240503 08:18:12.307924 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:12 dist-test-slave-3g67 krb5kdc[26784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:12.423992 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:12.500118 26797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:12.500299 26797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:12.500332 26797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:12.501626 26797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:12.501668 26797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:12.501688 26797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:12.501708 26797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:12.501729 26797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:12.503317 26797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:12.503530 26797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:12.503736 26797 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:12.507897 26805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:12.508530 26802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:12.509006 26803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:12.509744 26797 server_base.cc:1034] running on GCE node
I20240503 08:18:12.509976 26797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:12.510212 26797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:12.511343 26797 hybrid_clock.cc:648] HybridClock initialized: now 1714724292511327 us; error 38 us; skew 500 ppm
May 03 08:18:12 dist-test-slave-3g67 krb5kdc[26784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:12.517571 26797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:12.517982 26797 webserver.cc:460] Webserver started at http://127.0.0.1:39619/ using document root <none> and password file <none>
I20240503 08:18:12.518158 26797 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:12.518201 26797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:12.518308 26797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:12.519301 26797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "049f1fa153df4d3c852440561824371f"
format_stamp: "Formatted at 2024-05-03 08:18:12 on dist-test-slave-3g67"
server_key: "8fde04165e36cfc58ffa8f5e2e9cd866"
server_key_iv: "d7ea205d54345fa77114a361c4342075"
server_key_version: "encryptionkey@0"
I20240503 08:18:12.519605 26797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "049f1fa153df4d3c852440561824371f"
format_stamp: "Formatted at 2024-05-03 08:18:12 on dist-test-slave-3g67"
server_key: "8fde04165e36cfc58ffa8f5e2e9cd866"
server_key_iv: "d7ea205d54345fa77114a361c4342075"
server_key_version: "encryptionkey@0"
I20240503 08:18:12.520805 26797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:12.521471 26812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:12.521636 26797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:12.521708 26797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "049f1fa153df4d3c852440561824371f"
format_stamp: "Formatted at 2024-05-03 08:18:12 on dist-test-slave-3g67"
server_key: "8fde04165e36cfc58ffa8f5e2e9cd866"
server_key_iv: "d7ea205d54345fa77114a361c4342075"
server_key_version: "encryptionkey@0"
I20240503 08:18:12.521767 26797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:12.549050 26797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:12.551054 26797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:12.551183 26797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:12.554797 26797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33509
I20240503 08:18:12.554828 26864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33509 every 8 connection(s)
I20240503 08:18:12.555135 26797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:12.555958 26865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:12.557974 26865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f: Bootstrap starting.
I20240503 08:18:12.558229 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 26797
I20240503 08:18:12.558320 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:12.558480 6419 external_mini_cluster.cc:1428] Setting key a5f42e3c741ce5efa5d0a57404b6f24c
I20240503 08:18:12.558555 26865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:12.558853 26865 log.cc:824] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:12.559521 26865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f: No bootstrap required, opened a new log
I20240503 08:18:12.560958 26865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049f1fa153df4d3c852440561824371f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } }
I20240503 08:18:12.561101 26865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:12.561132 26865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049f1fa153df4d3c852440561824371f, State: Initialized, Role: FOLLOWER
I20240503 08:18:12.561246 26865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049f1fa153df4d3c852440561824371f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } }
I20240503 08:18:12.561321 26865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:12.561383 26865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:12.561429 26865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:12.562050 26865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049f1fa153df4d3c852440561824371f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } }
I20240503 08:18:12.562151 26865 leader_election.cc:302] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 049f1fa153df4d3c852440561824371f; no voters:
I20240503 08:18:12.562294 26865 leader_election.cc:288] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:12.562325 26870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:12.562430 26870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [term 1 LEADER]: Becoming Leader. State: Replica: 049f1fa153df4d3c852440561824371f, State: Running, Role: LEADER
I20240503 08:18:12.562526 26865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:12.562511 26870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049f1fa153df4d3c852440561824371f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } }
I20240503 08:18:12.562908 26871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "049f1fa153df4d3c852440561824371f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049f1fa153df4d3c852440561824371f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } } }
I20240503 08:18:12.563004 26871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:12.563220 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:12.558957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36209 (local address 127.0.0.1:33509)
0503 08:18:12.559130 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:12.559138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:12.559157 (+ 19us) server_negotiation.cc:407] Connection header received
0503 08:18:12.559565 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:12.559572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:12.559645 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:12.559784 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:12.560138 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:12.560849 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:12.561509 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:12.561762 (+ 253us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:12.562236 (+ 474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:12.562245 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:12.562245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:12.562408 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:12.562433 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:12.562442 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:12.562761 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:12.562832 (+ 71us) server_negotiation.cc:299] Negotiation successful
0503 08:18:12.562854 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20240503 08:18:12.563380 26872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 049f1fa153df4d3c852440561824371f. Latest consensus state: current_term: 1 leader_uuid: "049f1fa153df4d3c852440561824371f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049f1fa153df4d3c852440561824371f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } } }
I20240503 08:18:12.563455 26872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:12.563660 26876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:12.564090 26876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:12.564188 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:12.565521 26876 catalog_manager.cc:1347] Generated new cluster ID: a2f8bceafa984049b186a3c460a40366
I20240503 08:18:12.565579 26876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:12.568043 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:12.565453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45913 (local address 127.0.0.1:33509)
0503 08:18:12.565494 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:12.565496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:12.565679 (+ 183us) server_negotiation.cc:407] Connection header received
0503 08:18:12.565748 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:12.565749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:12.565757 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:12.565876 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:12.566211 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:12.566623 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:12.567099 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:12.567278 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:12.567771 (+ 493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:12.567774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:12.567774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:12.567778 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:12.567792 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:12.567795 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:12.567855 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:12.567927 (+ 72us) server_negotiation.cc:299] Negotiation successful
0503 08:18:12.567937 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240503 08:18:12.598803 26876 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:12.599539 26876 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:12.614991 26876 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 049f1fa153df4d3c852440561824371f: Generated new TSK 0
I20240503 08:18:12.615195 26876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:12.620468 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 26797
W20240503 08:18:12.624965 26782 connection.cc:535] client connection to 127.0.0.1:33509 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33509): Connection reset by peer (error 104)
2024-05-03T08:18:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:12 dist-test-slave-3g67 krb5kdc[26896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:12 dist-test-slave-3g67 krb5kdc[26896](info): set up 2 sockets
May 03 08:18:12 dist-test-slave-3g67 krb5kdc[26896](info): commencing operation
krb5kdc: starting...
W20240503 08:18:13.572060 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:13 dist-test-slave-3g67 krb5kdc[26896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:13.688457 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:13.767536 26909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:13.767740 26909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:13.767762 26909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:13.768990 26909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:13.769027 26909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:13.769042 26909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:13.769052 26909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:13.769063 26909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:13.770571 26909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:13.770725 26909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:13.770856 26909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:13.775349 26914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:13.775902 26917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:13.776412 26915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:13.777055 26909 server_base.cc:1034] running on GCE node
I20240503 08:18:13.777292 26909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:13.777514 26909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:13.778666 26909 hybrid_clock.cc:648] HybridClock initialized: now 1714724293778649 us; error 49 us; skew 500 ppm
May 03 08:18:13 dist-test-slave-3g67 krb5kdc[26896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:13.780982 26909 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:13.781348 26909 webserver.cc:460] Webserver started at http://127.0.0.1:32837/ using document root <none> and password file <none>
I20240503 08:18:13.781491 26909 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:13.781525 26909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:13.781602 26909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:13.782565 26909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "53c3c25eb24b4ab5aae7314a01557f19"
format_stamp: "Formatted at 2024-05-03 08:18:13 on dist-test-slave-3g67"
server_key: "3bda291eb8d8fbe78d9cd498e63b0042"
server_key_iv: "5ae9cbb0fc032e7e5a36f62042bd1dd8"
server_key_version: "encryptionkey@0"
I20240503 08:18:13.782819 26909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "53c3c25eb24b4ab5aae7314a01557f19"
format_stamp: "Formatted at 2024-05-03 08:18:13 on dist-test-slave-3g67"
server_key: "3bda291eb8d8fbe78d9cd498e63b0042"
server_key_iv: "5ae9cbb0fc032e7e5a36f62042bd1dd8"
server_key_version: "encryptionkey@0"
I20240503 08:18:13.783917 26909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:13.784518 26924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:13.784669 26909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:13.784737 26909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "53c3c25eb24b4ab5aae7314a01557f19"
format_stamp: "Formatted at 2024-05-03 08:18:13 on dist-test-slave-3g67"
server_key: "3bda291eb8d8fbe78d9cd498e63b0042"
server_key_iv: "5ae9cbb0fc032e7e5a36f62042bd1dd8"
server_key_version: "encryptionkey@0"
I20240503 08:18:13.784797 26909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:13.788100 26909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:13.790035 26909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:13.790123 26909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:13.793581 26909 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41619
I20240503 08:18:13.793624 26976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41619 every 8 connection(s)
I20240503 08:18:13.793879 26909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:13.794661 26977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:13.796706 26977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19: Bootstrap starting.
I20240503 08:18:13.797231 26977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:13.797462 26977 log.cc:824] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:13.798044 26977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19: No bootstrap required, opened a new log
I20240503 08:18:13.799381 26977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3c25eb24b4ab5aae7314a01557f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } }
I20240503 08:18:13.799492 26977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:13.799510 26977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c3c25eb24b4ab5aae7314a01557f19, State: Initialized, Role: FOLLOWER
I20240503 08:18:13.799651 26977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3c25eb24b4ab5aae7314a01557f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } }
I20240503 08:18:13.799739 26977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:13.799783 26977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:13.799829 26977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:13.800331 26977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3c25eb24b4ab5aae7314a01557f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } }
I20240503 08:18:13.800442 26977 leader_election.cc:302] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53c3c25eb24b4ab5aae7314a01557f19; no voters:
I20240503 08:18:13.800593 26977 leader_election.cc:288] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:13.800642 26980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:13.800786 26980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [term 1 LEADER]: Becoming Leader. State: Replica: 53c3c25eb24b4ab5aae7314a01557f19, State: Running, Role: LEADER
I20240503 08:18:13.800805 26977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:13.800918 26980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3c25eb24b4ab5aae7314a01557f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } }
I20240503 08:18:13.801239 26982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c3c25eb24b4ab5aae7314a01557f19. Latest consensus state: current_term: 1 leader_uuid: "53c3c25eb24b4ab5aae7314a01557f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3c25eb24b4ab5aae7314a01557f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } } }
I20240503 08:18:13.801347 26981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53c3c25eb24b4ab5aae7314a01557f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c3c25eb24b4ab5aae7314a01557f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41619 } } }
I20240503 08:18:13.801383 26982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:13.801434 26981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:13.801833 26989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:13.802260 26989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:13.802681 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 26909
I20240503 08:18:13.802773 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:13.802961 6419 external_mini_cluster.cc:1428] Setting key 11f0033492f2d1cda7b6feb2cc112a68
I20240503 08:18:13.803997 26989 catalog_manager.cc:1347] Generated new cluster ID: 66d534cbdfb146899c8c7e5af0984356
I20240503 08:18:13.804052 26989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:13.808125 26999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:13.803390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41993 (local address 127.0.0.1:41619)
0503 08:18:13.803715 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:13.803726 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:13.803750 (+ 24us) server_negotiation.cc:407] Connection header received
0503 08:18:13.803871 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:13.803877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:13.803906 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:13.804070 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:13.807775 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:13.807778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:13.807778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:13.807781 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:13.807798 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:13.807803 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:13.807877 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:13.807970 (+ 93us) server_negotiation.cc:299] Negotiation successful
0503 08:18:13.808002 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":83,"threads_started":1}
I20240503 08:18:13.808898 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:13.820889 26999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:13.810126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60123 (local address 127.0.0.1:41619)
0503 08:18:13.810165 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:13.810167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:13.819838 (+ 9671us) server_negotiation.cc:407] Connection header received
0503 08:18:13.819917 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:13.819918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:13.819926 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:13.819982 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:13.820632 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:13.820634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:13.820634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:13.820638 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:13.820654 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:13.820658 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:13.820715 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:13.820779 (+ 64us) server_negotiation.cc:299] Negotiation successful
0503 08:18:13.820797 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:18:13.822002 26989 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:13.822137 26989 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:13.832701 26989 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 53c3c25eb24b4ab5aae7314a01557f19: Generated new TSK 0
I20240503 08:18:13.832872 26989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:13.835891 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 26909
2024-05-03T08:18:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:13 dist-test-slave-3g67 krb5kdc[27008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:13 dist-test-slave-3g67 krb5kdc[27008](info): set up 2 sockets
May 03 08:18:13 dist-test-slave-3g67 krb5kdc[27008](info): commencing operation
krb5kdc: starting...
W20240503 08:18:14.788095 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:14 dist-test-slave-3g67 krb5kdc[27008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:14.900858 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:14.978637 27021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:14.978837 27021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:14.978859 27021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:14.980134 27021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:14.980172 27021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:14.980185 27021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:14.980198 27021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:14.980209 27021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:14.981787 27021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:14.981954 27021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:14.982096 27021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:14.986831 27026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:14.987161 27029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:14.987825 27027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:14.988364 27021 server_base.cc:1034] running on GCE node
I20240503 08:18:14.988605 27021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:14.988847 27021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:14.989984 27021 hybrid_clock.cc:648] HybridClock initialized: now 1714724294989967 us; error 33 us; skew 500 ppm
May 03 08:18:14 dist-test-slave-3g67 krb5kdc[27008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:14.992362 27021 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:14.992800 27021 webserver.cc:460] Webserver started at http://127.0.0.1:41129/ using document root <none> and password file <none>
I20240503 08:18:14.992981 27021 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:14.993026 27021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:14.993132 27021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:14.994097 27021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "809053a0e9484373b09d6667cc72c656"
format_stamp: "Formatted at 2024-05-03 08:18:14 on dist-test-slave-3g67"
server_key: "885c22644222b609ba08ded8c6845c9e"
server_key_iv: "4d701864799009fe7ec47fffac0d5b08"
server_key_version: "encryptionkey@0"
I20240503 08:18:14.994402 27021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "809053a0e9484373b09d6667cc72c656"
format_stamp: "Formatted at 2024-05-03 08:18:14 on dist-test-slave-3g67"
server_key: "885c22644222b609ba08ded8c6845c9e"
server_key_iv: "4d701864799009fe7ec47fffac0d5b08"
server_key_version: "encryptionkey@0"
I20240503 08:18:14.995605 27021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:18:14.996309 27036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:14.996469 27021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:14.996542 27021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "809053a0e9484373b09d6667cc72c656"
format_stamp: "Formatted at 2024-05-03 08:18:14 on dist-test-slave-3g67"
server_key: "885c22644222b609ba08ded8c6845c9e"
server_key_iv: "4d701864799009fe7ec47fffac0d5b08"
server_key_version: "encryptionkey@0"
I20240503 08:18:14.996603 27021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:15.008831 27021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:15.010615 27021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:15.010732 27021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:15.014144 27021 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44763
I20240503 08:18:15.014176 27088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44763 every 8 connection(s)
I20240503 08:18:15.014475 27021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:15.015136 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 27021
I20240503 08:18:15.015249 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:15.015250 27089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:15.015453 6419 external_mini_cluster.cc:1428] Setting key a276084e68089c239022f4f2ecae76b4
I20240503 08:18:15.018191 27089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656: Bootstrap starting.
I20240503 08:18:15.018782 27089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:15.019042 27089 log.cc:824] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:15.019826 27089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656: No bootstrap required, opened a new log
May 03 08:18:15 dist-test-slave-3g67 krb5kdc[27008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240503 08:18:15.021497 27089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809053a0e9484373b09d6667cc72c656" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20240503 08:18:15.021615 27089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:15.021646 27089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809053a0e9484373b09d6667cc72c656, State: Initialized, Role: FOLLOWER
I20240503 08:18:15.021744 27089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809053a0e9484373b09d6667cc72c656" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20240503 08:18:15.021823 27089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:15.021857 27089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:15.021907 27089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:15.022504 27089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809053a0e9484373b09d6667cc72c656" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20240503 08:18:15.022609 27089 leader_election.cc:302] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809053a0e9484373b09d6667cc72c656; no voters:
I20240503 08:18:15.022776 27089 leader_election.cc:288] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:15.022835 27094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:15.023013 27094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [term 1 LEADER]: Becoming Leader. State: Replica: 809053a0e9484373b09d6667cc72c656, State: Running, Role: LEADER
I20240503 08:18:15.023012 27089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:15.023169 27094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809053a0e9484373b09d6667cc72c656" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } }
I20240503 08:18:15.023459 27094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 809053a0e9484373b09d6667cc72c656. Latest consensus state: current_term: 1 leader_uuid: "809053a0e9484373b09d6667cc72c656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809053a0e9484373b09d6667cc72c656" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } } }
I20240503 08:18:15.023473 27095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "809053a0e9484373b09d6667cc72c656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809053a0e9484373b09d6667cc72c656" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44763 } } }
I20240503 08:18:15.023561 27094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:15.023571 27095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:15.023784 27098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:15.024278 27098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:15.025336 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:15.016005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43705 (local address 127.0.0.1:44763)
0503 08:18:15.016170 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:15.016181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:15.016202 (+ 21us) server_negotiation.cc:407] Connection header received
0503 08:18:15.016672 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:15.016681 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:15.016737 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:15.016900 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:15.017621 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:15.018724 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:15.019303 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:15.019552 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:15.021571 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:15.021578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:15.021578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:15.021593 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:15.023798 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:15.024192 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:15.024195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:15.024196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:15.024250 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:15.024459 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:15.024460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:15.024461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:15.024764 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:15.024933 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:15.025010 (+ 77us) server_negotiation.cc:299] Negotiation successful
0503 08:18:15.025131 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20240503 08:18:15.025857 27098 catalog_manager.cc:1347] Generated new cluster ID: a4641f3f91ce4d508ac531cb38211c91
I20240503 08:18:15.025911 27098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:15.026228 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:15.032318 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:15.027368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44891 (local address 127.0.0.1:44763)
0503 08:18:15.027412 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:15.027414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:15.027544 (+ 130us) server_negotiation.cc:407] Connection header received
0503 08:18:15.027601 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:15.027602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:15.027611 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:15.027674 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:15.028256 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:15.028682 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:15.029152 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:15.029333 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:15.030176 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:15.030179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:15.030179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:15.030185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:15.031342 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:15.031710 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:15.031711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:15.031712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:15.031732 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:15.031864 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:15.031865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:15.031865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:15.031944 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:15.032052 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:15.032138 (+ 86us) server_negotiation.cc:299] Negotiation successful
0503 08:18:15.032216 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240503 08:18:15.032565 27098 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:15.033074 27098 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:15.043347 27098 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 809053a0e9484373b09d6667cc72c656: Generated new TSK 0
I20240503 08:18:15.043531 27098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:15.046348 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 27021
2024-05-03T08:18:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:15 dist-test-slave-3g67 krb5kdc[27120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:15 dist-test-slave-3g67 krb5kdc[27120](info): set up 2 sockets
May 03 08:18:15 dist-test-slave-3g67 krb5kdc[27120](info): commencing operation
krb5kdc: starting...
W20240503 08:18:16.008219 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:16 dist-test-slave-3g67 krb5kdc[27120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:16.118366 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:16.200421 27133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:16.200614 27133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:16.200636 27133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:16.201884 27133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:16.201920 27133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:16.201933 27133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:16.201946 27133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:16.201962 27133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:16.203596 27133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:16.203778 27133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:16.203923 27133 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:16.208139 27141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:16.208689 27138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:16.209296 27139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:16.209854 27133 server_base.cc:1034] running on GCE node
I20240503 08:18:16.210078 27133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:16.210304 27133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:16.211443 27133 hybrid_clock.cc:648] HybridClock initialized: now 1714724296211424 us; error 37 us; skew 500 ppm
May 03 08:18:16 dist-test-slave-3g67 krb5kdc[27120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:16.213985 27133 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:16.214408 27133 webserver.cc:460] Webserver started at http://127.0.0.1:43259/ using document root <none> and password file <none>
I20240503 08:18:16.214599 27133 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:16.214646 27133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:16.214780 27133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:16.215865 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "b080cde9d82a44a3b3608c2d6b137fd7"
format_stamp: "Formatted at 2024-05-03 08:18:16 on dist-test-slave-3g67"
server_key: "d744ac89a9dfc9a70c8b47fbb139f2e0"
server_key_iv: "37509813b0ce1b972c67fb3c6ed0de4e"
server_key_version: "encryptionkey@0"
I20240503 08:18:16.216193 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "b080cde9d82a44a3b3608c2d6b137fd7"
format_stamp: "Formatted at 2024-05-03 08:18:16 on dist-test-slave-3g67"
server_key: "d744ac89a9dfc9a70c8b47fbb139f2e0"
server_key_iv: "37509813b0ce1b972c67fb3c6ed0de4e"
server_key_version: "encryptionkey@0"
I20240503 08:18:16.217522 27133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:16.218281 27148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:16.218528 27133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:16.218614 27133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "b080cde9d82a44a3b3608c2d6b137fd7"
format_stamp: "Formatted at 2024-05-03 08:18:16 on dist-test-slave-3g67"
server_key: "d744ac89a9dfc9a70c8b47fbb139f2e0"
server_key_iv: "37509813b0ce1b972c67fb3c6ed0de4e"
server_key_version: "encryptionkey@0"
I20240503 08:18:16.218683 27133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:16.247855 27133 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:16.249744 27133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:16.249902 27133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:16.253471 27133 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46121
I20240503 08:18:16.253499 27200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46121 every 8 connection(s)
I20240503 08:18:16.253799 27133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:16.254495 27201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:16.256520 27201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7: Bootstrap starting.
I20240503 08:18:16.257050 27201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:16.257295 27201 log.cc:824] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:16.257900 27201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7: No bootstrap required, opened a new log
I20240503 08:18:16.259207 27201 raft_consensus.cc:348] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b080cde9d82a44a3b3608c2d6b137fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } }
I20240503 08:18:16.259320 27201 raft_consensus.cc:374] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:16.259337 27201 raft_consensus.cc:729] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b080cde9d82a44a3b3608c2d6b137fd7, State: Initialized, Role: FOLLOWER
I20240503 08:18:16.259451 27201 consensus_queue.cc:260] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b080cde9d82a44a3b3608c2d6b137fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } }
I20240503 08:18:16.259510 27201 raft_consensus.cc:388] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:16.259531 27201 raft_consensus.cc:482] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:16.259554 27201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:16.260093 27201 raft_consensus.cc:504] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b080cde9d82a44a3b3608c2d6b137fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } }
I20240503 08:18:16.260177 27201 leader_election.cc:302] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b080cde9d82a44a3b3608c2d6b137fd7; no voters:
I20240503 08:18:16.260313 27201 leader_election.cc:288] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:16.260360 27204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:16.260527 27204 raft_consensus.cc:686] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [term 1 LEADER]: Becoming Leader. State: Replica: b080cde9d82a44a3b3608c2d6b137fd7, State: Running, Role: LEADER
I20240503 08:18:16.260625 27204 consensus_queue.cc:237] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b080cde9d82a44a3b3608c2d6b137fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } }
I20240503 08:18:16.260870 27201 sys_catalog.cc:564] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:16.260916 27205 sys_catalog.cc:455] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b080cde9d82a44a3b3608c2d6b137fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b080cde9d82a44a3b3608c2d6b137fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } } }
I20240503 08:18:16.260999 27205 sys_catalog.cc:458] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:16.261206 27206 sys_catalog.cc:455] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b080cde9d82a44a3b3608c2d6b137fd7. Latest consensus state: current_term: 1 leader_uuid: "b080cde9d82a44a3b3608c2d6b137fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b080cde9d82a44a3b3608c2d6b137fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } } }
I20240503 08:18:16.261240 27208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:16.261308 27206 sys_catalog.cc:458] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:16.261790 27208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:16.262946 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 27133
I20240503 08:18:16.263060 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:16.263177 27208 catalog_manager.cc:1347] Generated new cluster ID: 9d7230af2c07489d9a53cae8d70fb1c5
I20240503 08:18:16.263231 27208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:16.263259 6419 external_mini_cluster.cc:1428] Setting key fd6e86a383f5e38d26a16dd19b13d8ca
I20240503 08:18:16.276017 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:16.263859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44367 (local address 127.0.0.1:46121)
0503 08:18:16.271749 (+ 7890us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:16.271768 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:16.271796 (+ 28us) server_negotiation.cc:407] Connection header received
0503 08:18:16.271916 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:16.271922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:16.271969 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:16.272109 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:16.272584 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:16.273701 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:16.274491 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:16.274698 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:16.275320 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:16.275327 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:16.275327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:16.275523 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:16.275544 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:16.275551 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:16.275649 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:16.275705 (+ 56us) server_negotiation.cc:299] Negotiation successful
0503 08:18:16.275750 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7839,"thread_start_us":66,"threads_started":1}
I20240503 08:18:16.276841 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:16.278821 27208 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:16.279664 27208 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:16.281694 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:16.278396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38681 (local address 127.0.0.1:46121)
0503 08:18:16.278585 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:16.278587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:16.278595 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:18:16.278615 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:16.278616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:16.278627 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:16.278791 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:16.279207 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:16.279845 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:16.280556 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:16.280734 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:16.281444 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:16.281447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:16.281448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:16.281452 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:16.281467 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:16.281470 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:16.281495 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:16.281585 (+ 90us) server_negotiation.cc:299] Negotiation successful
0503 08:18:16.281623 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20240503 08:18:16.285749 27208 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b080cde9d82a44a3b3608c2d6b137fd7: Generated new TSK 0
I20240503 08:18:16.285930 27208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:16.289306 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 27133
2024-05-03T08:18:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:16 dist-test-slave-3g67 krb5kdc[27232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:16 dist-test-slave-3g67 krb5kdc[27232](info): set up 2 sockets
May 03 08:18:16 dist-test-slave-3g67 krb5kdc[27232](info): commencing operation
krb5kdc: starting...
W20240503 08:18:17.232074 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:17 dist-test-slave-3g67 krb5kdc[27232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:17.350381 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:17.430013 27245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:17.430210 27245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:17.430234 27245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:17.431486 27245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:17.431528 27245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:17.431543 27245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:17.431558 27245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:17.431571 27245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:17.433131 27245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:17.433346 27245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:17.433502 27245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:17.438130 27253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:17.438673 27251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:17.439210 27250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:17.439810 27245 server_base.cc:1034] running on GCE node
I20240503 08:18:17.440057 27245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:17.440296 27245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:17.441455 27245 hybrid_clock.cc:648] HybridClock initialized: now 1714724297441432 us; error 38 us; skew 500 ppm
May 03 08:18:17 dist-test-slave-3g67 krb5kdc[27232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:17.443884 27245 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:17.444245 27245 webserver.cc:460] Webserver started at http://127.0.0.1:33267/ using document root <none> and password file <none>
I20240503 08:18:17.444423 27245 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:17.444468 27245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:17.444571 27245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:17.445555 27245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "6ffb5f5886fe4184aa487aee7867b7ac"
format_stamp: "Formatted at 2024-05-03 08:18:17 on dist-test-slave-3g67"
server_key: "501f1025bad0b1c74db7e4c55178894f"
server_key_iv: "43ee1b8b7a35635d7f4f511b6eb20297"
server_key_version: "encryptionkey@0"
I20240503 08:18:17.445856 27245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "6ffb5f5886fe4184aa487aee7867b7ac"
format_stamp: "Formatted at 2024-05-03 08:18:17 on dist-test-slave-3g67"
server_key: "501f1025bad0b1c74db7e4c55178894f"
server_key_iv: "43ee1b8b7a35635d7f4f511b6eb20297"
server_key_version: "encryptionkey@0"
I20240503 08:18:17.447063 27245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:17.447782 27260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:17.447945 27245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:17.448016 27245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "6ffb5f5886fe4184aa487aee7867b7ac"
format_stamp: "Formatted at 2024-05-03 08:18:17 on dist-test-slave-3g67"
server_key: "501f1025bad0b1c74db7e4c55178894f"
server_key_iv: "43ee1b8b7a35635d7f4f511b6eb20297"
server_key_version: "encryptionkey@0"
I20240503 08:18:17.448083 27245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:17.451445 27245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:17.453204 27245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:17.453320 27245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:17.456876 27245 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38857
I20240503 08:18:17.456919 27312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38857 every 8 connection(s)
I20240503 08:18:17.457199 27245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:17.457991 27313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:17.460036 27313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac: Bootstrap starting.
I20240503 08:18:17.460570 27313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:17.460808 27313 log.cc:824] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:17.461483 27313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac: No bootstrap required, opened a new log
I20240503 08:18:17.462858 27313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb5f5886fe4184aa487aee7867b7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20240503 08:18:17.462975 27313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:17.462992 27313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ffb5f5886fe4184aa487aee7867b7ac, State: Initialized, Role: FOLLOWER
I20240503 08:18:17.463114 27313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb5f5886fe4184aa487aee7867b7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20240503 08:18:17.463184 27313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:17.463204 27313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:17.463229 27313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:17.463781 27313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb5f5886fe4184aa487aee7867b7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20240503 08:18:17.463866 27313 leader_election.cc:302] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ffb5f5886fe4184aa487aee7867b7ac; no voters:
I20240503 08:18:17.464002 27313 leader_election.cc:288] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:17.464046 27316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:17.464174 27316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [term 1 LEADER]: Becoming Leader. State: Replica: 6ffb5f5886fe4184aa487aee7867b7ac, State: Running, Role: LEADER
I20240503 08:18:17.464228 27313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:17.464298 27316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb5f5886fe4184aa487aee7867b7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20240503 08:18:17.464761 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 27245
I20240503 08:18:17.464740 27317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ffb5f5886fe4184aa487aee7867b7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb5f5886fe4184aa487aee7867b7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } } }
I20240503 08:18:17.464815 27317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:17.464857 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:17.464978 27318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ffb5f5886fe4184aa487aee7867b7ac. Latest consensus state: current_term: 1 leader_uuid: "6ffb5f5886fe4184aa487aee7867b7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb5f5886fe4184aa487aee7867b7ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } } }
I20240503 08:18:17.465025 27318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:17.465073 6419 external_mini_cluster.cc:1428] Setting key 7a353a0f90fa9bed679dceef7b52a365
I20240503 08:18:17.465256 27325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:17.465783 27325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:17.467172 27325 catalog_manager.cc:1347] Generated new cluster ID: 662a4586d1794958a1e40378e79c1e96
I20240503 08:18:17.467233 27325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:17.468189 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:17.465696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41413 (local address 127.0.0.1:38857)
0503 08:18:17.466378 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:17.466387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:17.466405 (+ 18us) server_negotiation.cc:407] Connection header received
0503 08:18:17.466486 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:17.466490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:17.466512 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:17.466631 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:17.467832 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:17.467835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:17.467835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:17.467840 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:17.467864 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:17.467871 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:17.467970 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:17.468003 (+ 33us) server_negotiation.cc:299] Negotiation successful
0503 08:18:17.468035 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":62,"threads_started":1}
I20240503 08:18:17.472352 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:17.474817 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:17.473740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42507 (local address 127.0.0.1:38857)
0503 08:18:17.473791 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:17.473793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:17.473801 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:18:17.473860 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:17.473861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:17.473866 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:17.473933 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:17.474585 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:17.474586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:17.474586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:17.474597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:17.474611 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:17.474615 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:17.474682 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:17.474722 (+ 40us) server_negotiation.cc:299] Negotiation successful
0503 08:18:17.474735 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240503 08:18:17.477288 27325 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:17.477420 27325 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:17.484298 27325 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6ffb5f5886fe4184aa487aee7867b7ac: Generated new TSK 0
I20240503 08:18:17.484478 27325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:17.491501 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 27245
2024-05-03T08:18:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:17 dist-test-slave-3g67 krb5kdc[27344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:17 dist-test-slave-3g67 krb5kdc[27344](info): set up 2 sockets
May 03 08:18:17 dist-test-slave-3g67 krb5kdc[27344](info): commencing operation
krb5kdc: starting...
W20240503 08:18:18.452071 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:18 dist-test-slave-3g67 krb5kdc[27344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:18.566159 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:18.644039 27357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:18.644234 27357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:18.644255 27357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:18.645565 27357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:18.645612 27357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:18.645625 27357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:18.645638 27357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:18.645650 27357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:18.647233 27357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:18.647423 27357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:18.647639 27357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:18.652020 27365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:18.652469 27362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:18.653123 27363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:18.653724 27357 server_base.cc:1034] running on GCE node
I20240503 08:18:18.653923 27357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:18.654124 27357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:18.655263 27357 hybrid_clock.cc:648] HybridClock initialized: now 1714724298655246 us; error 47 us; skew 500 ppm
May 03 08:18:18 dist-test-slave-3g67 krb5kdc[27344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:18.657611 27357 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:18.657996 27357 webserver.cc:460] Webserver started at http://127.0.0.1:36831/ using document root <none> and password file <none>
I20240503 08:18:18.658165 27357 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:18.658208 27357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:18.658335 27357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:18.659269 27357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "645e6ead9ace406faccacaa22e90a49e"
format_stamp: "Formatted at 2024-05-03 08:18:18 on dist-test-slave-3g67"
server_key: "40e755ecd7b1a346878c080dd16f1a51"
server_key_iv: "4c65fbd4d286b7fc9ff8e2ec7facee52"
server_key_version: "encryptionkey@0"
I20240503 08:18:18.659579 27357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "645e6ead9ace406faccacaa22e90a49e"
format_stamp: "Formatted at 2024-05-03 08:18:18 on dist-test-slave-3g67"
server_key: "40e755ecd7b1a346878c080dd16f1a51"
server_key_iv: "4c65fbd4d286b7fc9ff8e2ec7facee52"
server_key_version: "encryptionkey@0"
I20240503 08:18:18.660784 27357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:18:18.661438 27372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:18.661610 27357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:18.661684 27357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "645e6ead9ace406faccacaa22e90a49e"
format_stamp: "Formatted at 2024-05-03 08:18:18 on dist-test-slave-3g67"
server_key: "40e755ecd7b1a346878c080dd16f1a51"
server_key_iv: "4c65fbd4d286b7fc9ff8e2ec7facee52"
server_key_version: "encryptionkey@0"
I20240503 08:18:18.661742 27357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:18.684077 27357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:18.685959 27357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:18.686079 27357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:18.689730 27357 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36195
I20240503 08:18:18.689817 27424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36195 every 8 connection(s)
I20240503 08:18:18.690083 27357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:18.690271 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 27357
I20240503 08:18:18.690382 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:18.690549 6419 external_mini_cluster.cc:1428] Setting key 6acd7fc6fd9b896cada62227fb45307b
I20240503 08:18:18.690923 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:18.693912 27425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e: Bootstrap starting.
I20240503 08:18:18.694530 27425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:18.694792 27425 log.cc:824] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:18.695529 27425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e: No bootstrap required, opened a new log
May 03 08:18:18 dist-test-slave-3g67 krb5kdc[27344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240503 08:18:18.697378 27425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645e6ead9ace406faccacaa22e90a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20240503 08:18:18.697496 27425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:18.697513 27425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645e6ead9ace406faccacaa22e90a49e, State: Initialized, Role: FOLLOWER
I20240503 08:18:18.697599 27425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645e6ead9ace406faccacaa22e90a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20240503 08:18:18.697664 27425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:18.697700 27425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:18.697742 27425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:18.698464 27425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645e6ead9ace406faccacaa22e90a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20240503 08:18:18.698576 27425 leader_election.cc:302] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 645e6ead9ace406faccacaa22e90a49e; no voters:
I20240503 08:18:18.698745 27425 leader_election.cc:288] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:18.698791 27430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:18.698932 27430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [term 1 LEADER]: Becoming Leader. State: Replica: 645e6ead9ace406faccacaa22e90a49e, State: Running, Role: LEADER
I20240503 08:18:18.698971 27425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:18.699045 27430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645e6ead9ace406faccacaa22e90a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20240503 08:18:18.699396 27430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 645e6ead9ace406faccacaa22e90a49e. Latest consensus state: current_term: 1 leader_uuid: "645e6ead9ace406faccacaa22e90a49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645e6ead9ace406faccacaa22e90a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } } }
I20240503 08:18:18.699506 27430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:18.699414 27431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "645e6ead9ace406faccacaa22e90a49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645e6ead9ace406faccacaa22e90a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } } }
I20240503 08:18:18.699847 27431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:18.699985 27437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:18.700455 27437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:18.700752 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:18.691000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:49319 (local address 127.0.0.1:36195)
0503 08:18:18.691156 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:18.691165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:18.691185 (+ 20us) server_negotiation.cc:407] Connection header received
0503 08:18:18.691609 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:18.691614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:18.691687 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:18.691801 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:18.692753 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:18.693439 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:18.694187 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:18.694346 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:18.696861 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:18.696868 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:18.696868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:18.696883 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:18.698890 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:18.699363 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:18.699366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:18.699366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:18.699428 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:18.699702 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:18.699704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:18.699704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:18.699942 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:18.700111 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:18.700430 (+ 319us) server_negotiation.cc:299] Negotiation successful
0503 08:18:18.700535 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20240503 08:18:18.701993 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:18.702250 27437 catalog_manager.cc:1347] Generated new cluster ID: ee89af4b08ba48cc9b5f95e175d04a53
I20240503 08:18:18.702299 27437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:18.708621 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:18.703299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:36533 (local address 127.0.0.1:36195)
0503 08:18:18.703335 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:18.703337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:18.703346 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:18.703414 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:18.703414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:18.703424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:18.703477 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:18.704149 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:18.704618 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:18.705105 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:18.705299 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:18.706188 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:18.706193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:18.706193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:18.706202 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:18.707520 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:18.707874 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:18.707875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:18.707875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:18.707895 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:18.708068 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:18.708068 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:18.708069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:18.708170 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:18.708275 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:18.708456 (+ 181us) server_negotiation.cc:299] Negotiation successful
0503 08:18:18.708515 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:18:18.710383 27437 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:18.711166 27437 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:18.718422 27437 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 645e6ead9ace406faccacaa22e90a49e: Generated new TSK 0
I20240503 08:18:18.718617 27437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:18.728334 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 27357
W20240503 08:18:18.733101 27342 connection.cc:535] client connection to 127.0.0.1:36195 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36195): Connection reset by peer (error 104)
2024-05-03T08:18:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1237 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:18 dist-test-slave-3g67 krb5kdc[27456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:18 dist-test-slave-3g67 krb5kdc[27456](info): set up 2 sockets
May 03 08:18:18 dist-test-slave-3g67 krb5kdc[27456](info): commencing operation
krb5kdc: starting...
W20240503 08:18:19.672142 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:19 dist-test-slave-3g67 krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:19.786788 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:19.864421 27469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:19.864617 27469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:19.864639 27469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:19.865847 27469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:19.865882 27469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:19.865895 27469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:19.865908 27469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:19.865919 27469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:19.867453 27469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:19.867614 27469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:19.867785 27469 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:19.872303 27474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:19.872903 27477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:19.873448 27475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:19.874083 27469 server_base.cc:1034] running on GCE node
I20240503 08:18:19.874277 27469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:19.874465 27469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:19.875604 27469 hybrid_clock.cc:648] HybridClock initialized: now 1714724299875585 us; error 31 us; skew 500 ppm
May 03 08:18:19 dist-test-slave-3g67 krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:19.877928 27469 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:19.878286 27469 webserver.cc:460] Webserver started at http://127.0.0.1:39639/ using document root <none> and password file <none>
I20240503 08:18:19.878449 27469 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:19.878492 27469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:19.878635 27469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:19.879580 27469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "ae0add90e62a46baae3104c0b08f5b47"
format_stamp: "Formatted at 2024-05-03 08:18:19 on dist-test-slave-3g67"
server_key: "7ff6c661dc6198a5793a599a7400713b"
server_key_iv: "ed08bebe10c98374f33679891ad6649f"
server_key_version: "encryptionkey@0"
I20240503 08:18:19.879909 27469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "ae0add90e62a46baae3104c0b08f5b47"
format_stamp: "Formatted at 2024-05-03 08:18:19 on dist-test-slave-3g67"
server_key: "7ff6c661dc6198a5793a599a7400713b"
server_key_iv: "ed08bebe10c98374f33679891ad6649f"
server_key_version: "encryptionkey@0"
I20240503 08:18:19.881088 27469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:19.881701 27484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:19.881856 27469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:19.881929 27469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "ae0add90e62a46baae3104c0b08f5b47"
format_stamp: "Formatted at 2024-05-03 08:18:19 on dist-test-slave-3g67"
server_key: "7ff6c661dc6198a5793a599a7400713b"
server_key_iv: "ed08bebe10c98374f33679891ad6649f"
server_key_version: "encryptionkey@0"
I20240503 08:18:19.881986 27469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:19.899890 27469 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:19.901957 27469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:19.902079 27469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:19.905565 27469 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46457
I20240503 08:18:19.905602 27536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46457 every 8 connection(s)
I20240503 08:18:19.905891 27469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:19.906752 27537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:19.908872 27537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47: Bootstrap starting.
I20240503 08:18:19.909365 27537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:19.909590 27537 log.cc:824] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:19.910210 27537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47: No bootstrap required, opened a new log
I20240503 08:18:19.910980 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 27469
I20240503 08:18:19.911114 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:19.911293 6419 external_mini_cluster.cc:1428] Setting key 55dcec4bf64bb28f531073b05e2a5b11
I20240503 08:18:19.912189 27537 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0add90e62a46baae3104c0b08f5b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } }
I20240503 08:18:19.912315 27537 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:19.912343 27537 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0add90e62a46baae3104c0b08f5b47, State: Initialized, Role: FOLLOWER
I20240503 08:18:19.912480 27537 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0add90e62a46baae3104c0b08f5b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } }
I20240503 08:18:19.912552 27537 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:19.912580 27537 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:19.912627 27537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:19.913148 27537 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0add90e62a46baae3104c0b08f5b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } }
I20240503 08:18:19.913228 27537 leader_election.cc:302] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae0add90e62a46baae3104c0b08f5b47; no voters:
I20240503 08:18:19.913381 27537 leader_election.cc:288] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:19.913422 27542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:19.913539 27542 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [term 1 LEADER]: Becoming Leader. State: Replica: ae0add90e62a46baae3104c0b08f5b47, State: Running, Role: LEADER
I20240503 08:18:19.913610 27537 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:19.913668 27542 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0add90e62a46baae3104c0b08f5b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } }
I20240503 08:18:19.914043 27542 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae0add90e62a46baae3104c0b08f5b47. Latest consensus state: current_term: 1 leader_uuid: "ae0add90e62a46baae3104c0b08f5b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0add90e62a46baae3104c0b08f5b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } } }
I20240503 08:18:19.914064 27544 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae0add90e62a46baae3104c0b08f5b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0add90e62a46baae3104c0b08f5b47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } } }
I20240503 08:18:19.914218 27542 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:19.914224 27544 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47 [sys.catalog]: This master's current role is: LEADER
W20240503 08:18:19.915270 27558 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:18:19.915387 27558 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:18:19.915616 27551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:19.915817 27551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:19.916426 27541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:19.911845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:47329 (local address 127.0.0.1:46457)
0503 08:18:19.912018 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:19.912027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:19.912044 (+ 17us) server_negotiation.cc:407] Connection header received
0503 08:18:19.912152 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:19.912156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:19.912195 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:19.912302 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:19.912734 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:19.913969 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:19.914946 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:19.915257 (+ 311us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:19.915695 (+ 438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:19.915701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:19.915702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:19.915874 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:19.915898 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:19.915906 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:19.915998 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:19.916122 (+ 124us) server_negotiation.cc:299] Negotiation successful
0503 08:18:19.916259 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20240503 08:18:19.917142 27551 catalog_manager.cc:1347] Generated new cluster ID: f83cf741dcc445c798e8f411d81ebdee
I20240503 08:18:19.917227 27551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:19.917589 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:19.921768 27541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:19.918896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:38421 (local address 127.0.0.1:46457)
0503 08:18:19.918932 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:19.918934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:19.919096 (+ 162us) server_negotiation.cc:407] Connection header received
0503 08:18:19.919140 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:19.919141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:19.919152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:19.919217 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:19.919517 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:19.920226 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:19.920884 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:19.921063 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:19.921494 (+ 431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:19.921497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:19.921497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:19.921502 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:19.921516 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:19.921521 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:19.921606 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:19.921686 (+ 80us) server_negotiation.cc:299] Negotiation successful
0503 08:18:19.921695 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240503 08:18:19.934921 27551 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:19.935770 27551 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:19.945245 27551 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ae0add90e62a46baae3104c0b08f5b47: Generated new TSK 0
I20240503 08:18:19.945448 27551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:19.947546 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 27469
W20240503 08:18:19.952059 27454 connection.cc:535] client connection to 127.0.0.1:46457 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46457): Connection reset by peer (error 104)
2024-05-03T08:18:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:19 dist-test-slave-3g67 krb5kdc[27568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:20 dist-test-slave-3g67 krb5kdc[27568](info): set up 2 sockets
May 03 08:18:20 dist-test-slave-3g67 krb5kdc[27568](info): commencing operation
krb5kdc: starting...
W20240503 08:18:20.900056 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:20 dist-test-slave-3g67 krb5kdc[27568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:21.015192 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:21.092608 27581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:21.092803 27581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:21.092825 27581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:21.094065 27581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:21.094106 27581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:21.094120 27581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:21.094132 27581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:21.094143 27581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:21.095721 27581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:21.095896 27581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:21.096038 27581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:21.100570 27586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:21.101042 27587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:21.101680 27589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:21.102308 27581 server_base.cc:1034] running on GCE node
I20240503 08:18:21.102540 27581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:21.102770 27581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:21.103920 27581 hybrid_clock.cc:648] HybridClock initialized: now 1714724301103894 us; error 52 us; skew 500 ppm
May 03 08:18:21 dist-test-slave-3g67 krb5kdc[27568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:21.106555 27581 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:21.106943 27581 webserver.cc:460] Webserver started at http://127.0.0.1:34325/ using document root <none> and password file <none>
I20240503 08:18:21.107125 27581 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:21.107170 27581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:21.107275 27581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:21.108305 27581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "363334ff25894afbaa58faa0ef9c712b"
format_stamp: "Formatted at 2024-05-03 08:18:21 on dist-test-slave-3g67"
server_key: "b53f88f89b2c1f12476bf48912014afd"
server_key_iv: "dc97576140f707a9d13665ea0b2ba848"
server_key_version: "encryptionkey@0"
I20240503 08:18:21.108601 27581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "363334ff25894afbaa58faa0ef9c712b"
format_stamp: "Formatted at 2024-05-03 08:18:21 on dist-test-slave-3g67"
server_key: "b53f88f89b2c1f12476bf48912014afd"
server_key_iv: "dc97576140f707a9d13665ea0b2ba848"
server_key_version: "encryptionkey@0"
I20240503 08:18:21.109822 27581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:21.110484 27596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:21.110654 27581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:21.110728 27581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "363334ff25894afbaa58faa0ef9c712b"
format_stamp: "Formatted at 2024-05-03 08:18:21 on dist-test-slave-3g67"
server_key: "b53f88f89b2c1f12476bf48912014afd"
server_key_iv: "dc97576140f707a9d13665ea0b2ba848"
server_key_version: "encryptionkey@0"
I20240503 08:18:21.110788 27581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:21.114245 27581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:21.116070 27581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:21.116184 27581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:21.119675 27581 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39823
I20240503 08:18:21.119714 27648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39823 every 8 connection(s)
I20240503 08:18:21.119987 27581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:21.120821 27649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:21.122901 27649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b: Bootstrap starting.
I20240503 08:18:21.123443 27649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:21.123734 27649 log.cc:824] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:21.124377 27649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b: No bootstrap required, opened a new log
I20240503 08:18:21.125779 27649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363334ff25894afbaa58faa0ef9c712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } }
I20240503 08:18:21.125895 27649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:21.125912 27649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363334ff25894afbaa58faa0ef9c712b, State: Initialized, Role: FOLLOWER
I20240503 08:18:21.126022 27649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363334ff25894afbaa58faa0ef9c712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } }
I20240503 08:18:21.126088 27649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:21.126109 27649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:21.126137 27649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:21.126669 27649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363334ff25894afbaa58faa0ef9c712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } }
I20240503 08:18:21.126751 27649 leader_election.cc:302] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 363334ff25894afbaa58faa0ef9c712b; no voters:
I20240503 08:18:21.126906 27649 leader_election.cc:288] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:21.126964 27652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:21.127108 27649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:21.127151 27652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [term 1 LEADER]: Becoming Leader. State: Replica: 363334ff25894afbaa58faa0ef9c712b, State: Running, Role: LEADER
I20240503 08:18:21.127254 27652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363334ff25894afbaa58faa0ef9c712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } }
I20240503 08:18:21.127627 27653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "363334ff25894afbaa58faa0ef9c712b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363334ff25894afbaa58faa0ef9c712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } } }
I20240503 08:18:21.127633 27654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 363334ff25894afbaa58faa0ef9c712b. Latest consensus state: current_term: 1 leader_uuid: "363334ff25894afbaa58faa0ef9c712b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363334ff25894afbaa58faa0ef9c712b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39823 } } }
I20240503 08:18:21.127775 27653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:21.127782 27654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:21.127969 27657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:21.128417 27657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:21.129470 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 27581
I20240503 08:18:21.129580 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:21.129781 6419 external_mini_cluster.cc:1428] Setting key 9f15a2d2b10635386d41dea3382b60d7
I20240503 08:18:21.130281 27657 catalog_manager.cc:1347] Generated new cluster ID: 1529c70d8dab4560b81e91e912d89d9d
I20240503 08:18:21.130332 27657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:21.137216 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:21.130247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:51927 (local address 127.0.0.1:39823)
0503 08:18:21.130403 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:21.130412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:21.135063 (+ 4651us) server_negotiation.cc:407] Connection header received
0503 08:18:21.135290 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:21.135296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:21.135325 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:21.135455 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:21.136679 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:21.136682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:21.136683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:21.136689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:21.136720 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:21.136729 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:21.136797 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:21.136975 (+ 178us) server_negotiation.cc:299] Negotiation successful
0503 08:18:21.137019 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20240503 08:18:21.138371 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:21.145812 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:21.144402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:51733 (local address 127.0.0.1:39823)
0503 08:18:21.144455 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:21.144458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:21.144722 (+ 264us) server_negotiation.cc:407] Connection header received
0503 08:18:21.144788 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:21.144789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:21.144795 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:21.144841 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:21.145560 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:21.145562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:21.145562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:21.145567 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:21.145581 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:21.145586 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:21.145614 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:21.145723 (+ 109us) server_negotiation.cc:299] Negotiation successful
0503 08:18:21.145740 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240503 08:18:21.147038 27657 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:21.147197 27657 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:21.153391 27657 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 363334ff25894afbaa58faa0ef9c712b: Generated new TSK 0
I20240503 08:18:21.153682 27657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:21.156668 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 27581
2024-05-03T08:18:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:21 dist-test-slave-3g67 krb5kdc[27680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:21 dist-test-slave-3g67 krb5kdc[27680](info): set up 2 sockets
May 03 08:18:21 dist-test-slave-3g67 krb5kdc[27680](info): commencing operation
krb5kdc: starting...
W20240503 08:18:22.112116 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:22 dist-test-slave-3g67 krb5kdc[27680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:22.230336 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:22.308346 27693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:22.308537 27693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:22.308560 27693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:22.309805 27693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:22.309847 27693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:22.309861 27693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:22.309872 27693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:22.309883 27693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:22.311494 27693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:22.311713 27693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:22.311911 27693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:22.316417 27699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:22.316851 27698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:22.317512 27701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:22.318064 27693 server_base.cc:1034] running on GCE node
I20240503 08:18:22.318302 27693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:22.318563 27693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:22.319670 27693 hybrid_clock.cc:648] HybridClock initialized: now 1714724302319653 us; error 36 us; skew 500 ppm
May 03 08:18:22 dist-test-slave-3g67 krb5kdc[27680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:22.322074 27693 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:22.322456 27693 webserver.cc:460] Webserver started at http://127.0.0.1:36685/ using document root <none> and password file <none>
I20240503 08:18:22.322633 27693 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:22.322677 27693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:22.322784 27693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:22.323856 27693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "e29ee1e6a5a14466a2fae8abc6a0b86c"
format_stamp: "Formatted at 2024-05-03 08:18:22 on dist-test-slave-3g67"
server_key: "6322b78173ec7826e69e055fa12cf0b4"
server_key_iv: "90c9560f5ad5722f0ad51995b9e13c55"
server_key_version: "encryptionkey@0"
I20240503 08:18:22.324158 27693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "e29ee1e6a5a14466a2fae8abc6a0b86c"
format_stamp: "Formatted at 2024-05-03 08:18:22 on dist-test-slave-3g67"
server_key: "6322b78173ec7826e69e055fa12cf0b4"
server_key_iv: "90c9560f5ad5722f0ad51995b9e13c55"
server_key_version: "encryptionkey@0"
I20240503 08:18:22.325415 27693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:22.326275 27708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:22.326478 27693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:22.326596 27693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "e29ee1e6a5a14466a2fae8abc6a0b86c"
format_stamp: "Formatted at 2024-05-03 08:18:22 on dist-test-slave-3g67"
server_key: "6322b78173ec7826e69e055fa12cf0b4"
server_key_iv: "90c9560f5ad5722f0ad51995b9e13c55"
server_key_version: "encryptionkey@0"
I20240503 08:18:22.326663 27693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:22.356686 27693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:22.358738 27693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:22.358861 27693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:22.362380 27693 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38165
I20240503 08:18:22.362435 27760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38165 every 8 connection(s)
I20240503 08:18:22.362727 27693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:22.363456 27761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:22.364869 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 27693
I20240503 08:18:22.364966 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:22.365124 6419 external_mini_cluster.cc:1428] Setting key 49089dab59c6520cccb42f758b06da9e
I20240503 08:18:22.365808 27761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c: Bootstrap starting.
I20240503 08:18:22.366439 27761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:22.366745 27761 log.cc:824] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:22.367362 27761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c: No bootstrap required, opened a new log
I20240503 08:18:22.369371 27761 raft_consensus.cc:348] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29ee1e6a5a14466a2fae8abc6a0b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38165 } }
I20240503 08:18:22.369524 27761 raft_consensus.cc:374] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:22.369571 27761 raft_consensus.cc:729] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e29ee1e6a5a14466a2fae8abc6a0b86c, State: Initialized, Role: FOLLOWER
I20240503 08:18:22.369683 27761 consensus_queue.cc:260] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29ee1e6a5a14466a2fae8abc6a0b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38165 } }
I20240503 08:18:22.369751 27761 raft_consensus.cc:388] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:22.369794 27761 raft_consensus.cc:482] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:22.369838 27761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:22.370311 27761 raft_consensus.cc:504] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29ee1e6a5a14466a2fae8abc6a0b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38165 } }
I20240503 08:18:22.370400 27761 leader_election.cc:302] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e29ee1e6a5a14466a2fae8abc6a0b86c; no voters:
May 03 08:18:22 dist-test-slave-3g67 krb5kdc[27680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240503 08:18:22.371071 27761 leader_election.cc:288] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:22.371122 27766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:22.371261 27766 raft_consensus.cc:686] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [term 1 LEADER]: Becoming Leader. State: Replica: e29ee1e6a5a14466a2fae8abc6a0b86c, State: Running, Role: LEADER
I20240503 08:18:22.371316 27761 sys_catalog.cc:564] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:22.371349 27766 consensus_queue.cc:237] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29ee1e6a5a14466a2fae8abc6a0b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38165 } }
I20240503 08:18:22.371656 27766 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e29ee1e6a5a14466a2fae8abc6a0b86c. Latest consensus state: current_term: 1 leader_uuid: "e29ee1e6a5a14466a2fae8abc6a0b86c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29ee1e6a5a14466a2fae8abc6a0b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38165 } } }
I20240503 08:18:22.371733 27766 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:22.371922 27771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:22.371665 27767 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e29ee1e6a5a14466a2fae8abc6a0b86c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29ee1e6a5a14466a2fae8abc6a0b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38165 } } }
I20240503 08:18:22.372154 27767 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:22.372377 27771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:22.374073 27771 catalog_manager.cc:1347] Generated new cluster ID: 2aa3b2f6dfe6409c8bb6fda7ce2d2978
I20240503 08:18:22.374131 27771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:22.375761 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:22.365659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:52685 (local address 127.0.0.1:38165)
0503 08:18:22.365837 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:22.365848 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:22.365870 (+ 22us) server_negotiation.cc:407] Connection header received
0503 08:18:22.366475 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:22.366484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:22.366540 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:22.366705 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:22.367400 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:22.368599 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:22.369158 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:22.369388 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:22.371443 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:22.371452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:22.371452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:22.371472 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:22.373965 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:22.374399 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:22.374402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:22.374402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:22.374459 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:22.374683 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:22.374684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:22.374684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:22.374895 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:22.375037 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:22.375459 (+ 422us) server_negotiation.cc:299] Negotiation successful
0503 08:18:22.375557 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20240503 08:18:22.376595 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:22.394657 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:22.387996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:45315 (local address 127.0.0.1:38165)
0503 08:18:22.388036 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:22.388038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:22.388191 (+ 153us) server_negotiation.cc:407] Connection header received
0503 08:18:22.388230 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:22.388231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:22.388244 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:22.388325 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:22.389128 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:22.389881 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:22.390553 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:22.390755 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:22.391851 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:22.391855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:22.391855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:22.391861 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:22.393511 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:22.393913 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:22.393914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:22.393915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:22.393936 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:22.394109 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:22.394110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:22.394110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:22.394211 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:22.394315 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:22.394474 (+ 159us) server_negotiation.cc:299] Negotiation successful
0503 08:18:22.394536 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:18:22.394765 27771 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:22.395340 27771 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:22.399498 27771 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e29ee1e6a5a14466a2fae8abc6a0b86c: Generated new TSK 0
I20240503 08:18:22.399682 27771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:22.404286 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 27693
W20240503 08:18:22.409035 27678 connection.cc:535] client connection to 127.0.0.1:38165 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38165): Connection reset by peer (error 104)
2024-05-03T08:18:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:22 dist-test-slave-3g67 krb5kdc[27792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:22 dist-test-slave-3g67 krb5kdc[27792](info): set up 2 sockets
May 03 08:18:22 dist-test-slave-3g67 krb5kdc[27792](info): commencing operation
krb5kdc: starting...
W20240503 08:18:23.363966 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:23 dist-test-slave-3g67 krb5kdc[27792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:23.480953 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:23.558689 27805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:23.558879 27805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:23.558914 27805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:23.560225 27805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:23.560266 27805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:23.560286 27805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:23.560307 27805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:23.560324 27805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:23.561918 27805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:23.562122 27805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:23.562294 27805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:23.566663 27810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:23.567261 27811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:23.567780 27813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:23.568497 27805 server_base.cc:1034] running on GCE node
I20240503 08:18:23.568698 27805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:23.568910 27805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:23.570066 27805 hybrid_clock.cc:648] HybridClock initialized: now 1714724303570045 us; error 35 us; skew 500 ppm
May 03 08:18:23 dist-test-slave-3g67 krb5kdc[27792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:23.572469 27805 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:23.572873 27805 webserver.cc:460] Webserver started at http://127.0.0.1:35923/ using document root <none> and password file <none>
I20240503 08:18:23.573043 27805 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:23.573086 27805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:23.573202 27805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:23.574169 27805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "4eb1552b3f6843dc81bdf21a20972e13"
format_stamp: "Formatted at 2024-05-03 08:18:23 on dist-test-slave-3g67"
server_key: "8789d10d73a1998dfae3b32e2064a148"
server_key_iv: "035ffd262b69bef0bbb8e5684ba16c01"
server_key_version: "encryptionkey@0"
I20240503 08:18:23.574467 27805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "4eb1552b3f6843dc81bdf21a20972e13"
format_stamp: "Formatted at 2024-05-03 08:18:23 on dist-test-slave-3g67"
server_key: "8789d10d73a1998dfae3b32e2064a148"
server_key_iv: "035ffd262b69bef0bbb8e5684ba16c01"
server_key_version: "encryptionkey@0"
I20240503 08:18:23.575649 27805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:18:23.576354 27820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:23.576545 27805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:23.576628 27805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "4eb1552b3f6843dc81bdf21a20972e13"
format_stamp: "Formatted at 2024-05-03 08:18:23 on dist-test-slave-3g67"
server_key: "8789d10d73a1998dfae3b32e2064a148"
server_key_iv: "035ffd262b69bef0bbb8e5684ba16c01"
server_key_version: "encryptionkey@0"
I20240503 08:18:23.576704 27805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:23.587754 27805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:23.589532 27805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:23.589637 27805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:23.593119 27805 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40101
I20240503 08:18:23.593163 27872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40101 every 8 connection(s)
I20240503 08:18:23.593439 27805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:23.594182 27873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:23.594964 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 27805
I20240503 08:18:23.595062 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:23.595217 6419 external_mini_cluster.cc:1428] Setting key ada3fb27598bb3a7d0c999040a4e8b62
I20240503 08:18:23.596693 27873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13: Bootstrap starting.
I20240503 08:18:23.597294 27873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:23.597546 27873 log.cc:824] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:23.598207 27873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13: No bootstrap required, opened a new log
I20240503 08:18:23.599960 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:23.595694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:45589 (local address 127.0.0.1:40101)
0503 08:18:23.595847 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:23.595859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:23.595883 (+ 24us) server_negotiation.cc:407] Connection header received
0503 08:18:23.596502 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:23.596511 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:23.596580 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:23.596729 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:23.597055 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:23.598078 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:23.598693 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:23.598915 (+ 222us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:23.599259 (+ 344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:23.599266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:23.599266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:23.599436 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:23.599463 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:23.599471 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:23.599588 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:23.599716 (+ 128us) server_negotiation.cc:299] Negotiation successful
0503 08:18:23.599749 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20240503 08:18:23.600186 27873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb1552b3f6843dc81bdf21a20972e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20240503 08:18:23.600315 27873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:23.600342 27873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb1552b3f6843dc81bdf21a20972e13, State: Initialized, Role: FOLLOWER
I20240503 08:18:23.600432 27873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb1552b3f6843dc81bdf21a20972e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20240503 08:18:23.600493 27873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:23.600522 27873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:23.600567 27873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:23.600876 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:23.601187 27873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb1552b3f6843dc81bdf21a20972e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20240503 08:18:23.601294 27873 leader_election.cc:302] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eb1552b3f6843dc81bdf21a20972e13; no voters:
I20240503 08:18:23.601439 27873 leader_election.cc:288] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:23.601481 27878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:23.601611 27878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb1552b3f6843dc81bdf21a20972e13, State: Running, Role: LEADER
I20240503 08:18:23.601656 27873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:23.601697 27878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb1552b3f6843dc81bdf21a20972e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20240503 08:18:23.601991 27878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eb1552b3f6843dc81bdf21a20972e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb1552b3f6843dc81bdf21a20972e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } } }
I20240503 08:18:23.602063 27878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:23.602211 27878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eb1552b3f6843dc81bdf21a20972e13. Latest consensus state: current_term: 1 leader_uuid: "4eb1552b3f6843dc81bdf21a20972e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb1552b3f6843dc81bdf21a20972e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } } }
I20240503 08:18:23.602273 27878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:23.602281 27888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:23.602758 27888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:23.604705 27888 catalog_manager.cc:1347] Generated new cluster ID: bf0e19d740e74621a8f4dc84a84e2fd5
I20240503 08:18:23.604753 27888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:23.606132 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:23.603644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:57155 (local address 127.0.0.1:40101)
0503 08:18:23.603678 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:23.603680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:23.603895 (+ 215us) server_negotiation.cc:407] Connection header received
0503 08:18:23.603966 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:23.603967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:23.603976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:23.604031 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:23.604320 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:23.604774 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:23.605260 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:23.605434 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:23.605872 (+ 438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:23.605875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:23.605875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:23.605879 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:23.605892 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:23.605896 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:23.605959 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:23.606037 (+ 78us) server_negotiation.cc:299] Negotiation successful
0503 08:18:23.606046 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240503 08:18:23.622231 27888 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:23.622920 27888 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:23.634413 27888 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4eb1552b3f6843dc81bdf21a20972e13: Generated new TSK 0
I20240503 08:18:23.634590 27888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:23.643560 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 27805
W20240503 08:18:23.647931 27790 connection.cc:535] client connection to 127.0.0.1:40101 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40101): Connection reset by peer (error 104)
2024-05-03T08:18:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1238 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:23 dist-test-slave-3g67 krb5kdc[27904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:23 dist-test-slave-3g67 krb5kdc[27904](info): set up 2 sockets
May 03 08:18:23 dist-test-slave-3g67 krb5kdc[27904](info): commencing operation
krb5kdc: starting...
W20240503 08:18:24.604135 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:24 dist-test-slave-3g67 krb5kdc[27904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240503 08:18:24.714963 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:24.793015 27917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:24.793210 27917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:24.793232 27917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:24.794502 27917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:24.794540 27917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:24.794554 27917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:24.794567 27917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:24.794579 27917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:24.796165 27917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:24.796341 27917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:24.796479 27917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:24.801002 27925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:24.801491 27922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:24.802076 27923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:24.802723 27917 server_base.cc:1034] running on GCE node
I20240503 08:18:24.802961 27917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:24.803221 27917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:24.804363 27917 hybrid_clock.cc:648] HybridClock initialized: now 1714724304804346 us; error 33 us; skew 500 ppm
May 03 08:18:24 dist-test-slave-3g67 krb5kdc[27904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:24.806737 27917 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240503 08:18:24.807137 27917 webserver.cc:460] Webserver started at http://127.0.0.1:44549/ using document root <none> and password file <none>
I20240503 08:18:24.807300 27917 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:24.807336 27917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:24.807420 27917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:24.808405 27917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "f91971a9fc0849f2a00aecc9b2d0bd9e"
format_stamp: "Formatted at 2024-05-03 08:18:24 on dist-test-slave-3g67"
server_key: "5212420b5354d20467dd3b4a6736dbe9"
server_key_iv: "cf5ffe72a90616968e257e631add0114"
server_key_version: "encryptionkey@0"
I20240503 08:18:24.808681 27917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "f91971a9fc0849f2a00aecc9b2d0bd9e"
format_stamp: "Formatted at 2024-05-03 08:18:24 on dist-test-slave-3g67"
server_key: "5212420b5354d20467dd3b4a6736dbe9"
server_key_iv: "cf5ffe72a90616968e257e631add0114"
server_key_version: "encryptionkey@0"
I20240503 08:18:24.809820 27917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:18:24.810424 27932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:24.810600 27917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:24.810674 27917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "f91971a9fc0849f2a00aecc9b2d0bd9e"
format_stamp: "Formatted at 2024-05-03 08:18:24 on dist-test-slave-3g67"
server_key: "5212420b5354d20467dd3b4a6736dbe9"
server_key_iv: "cf5ffe72a90616968e257e631add0114"
server_key_version: "encryptionkey@0"
I20240503 08:18:24.810734 27917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:24.814200 27917 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:24.816208 27917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:24.816325 27917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:24.819891 27917 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39343
I20240503 08:18:24.819938 27984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39343 every 8 connection(s)
I20240503 08:18:24.820204 27917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:24.821027 27985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:24.823403 27985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e: Bootstrap starting.
I20240503 08:18:24.823990 27985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:24.824226 27985 log.cc:824] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:24.824868 27985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e: No bootstrap required, opened a new log
I20240503 08:18:24.826278 27985 raft_consensus.cc:348] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91971a9fc0849f2a00aecc9b2d0bd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20240503 08:18:24.826392 27985 raft_consensus.cc:374] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:24.826408 27985 raft_consensus.cc:729] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91971a9fc0849f2a00aecc9b2d0bd9e, State: Initialized, Role: FOLLOWER
I20240503 08:18:24.826529 27985 consensus_queue.cc:260] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91971a9fc0849f2a00aecc9b2d0bd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20240503 08:18:24.826588 27985 raft_consensus.cc:388] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:24.826610 27985 raft_consensus.cc:482] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:24.826656 27985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:24.827181 27985 raft_consensus.cc:504] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91971a9fc0849f2a00aecc9b2d0bd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20240503 08:18:24.827270 27985 leader_election.cc:302] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f91971a9fc0849f2a00aecc9b2d0bd9e; no voters:
I20240503 08:18:24.827404 27985 leader_election.cc:288] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:24.827477 27988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:24.827661 27985 sys_catalog.cc:564] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:24.827648 27988 raft_consensus.cc:686] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [term 1 LEADER]: Becoming Leader. State: Replica: f91971a9fc0849f2a00aecc9b2d0bd9e, State: Running, Role: LEADER
I20240503 08:18:24.827785 27988 consensus_queue.cc:237] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91971a9fc0849f2a00aecc9b2d0bd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20240503 08:18:24.828166 27990 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f91971a9fc0849f2a00aecc9b2d0bd9e. Latest consensus state: current_term: 1 leader_uuid: "f91971a9fc0849f2a00aecc9b2d0bd9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91971a9fc0849f2a00aecc9b2d0bd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } } }
I20240503 08:18:24.828255 27990 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:24.828521 27997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:24.828254 27989 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f91971a9fc0849f2a00aecc9b2d0bd9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91971a9fc0849f2a00aecc9b2d0bd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } } }
I20240503 08:18:24.828744 27989 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:24.828990 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 27917
I20240503 08:18:24.829064 27997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:24.829104 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:24.829254 6419 external_mini_cluster.cc:1428] Setting key 78386821797ef82e4df711604d1cf1c3
I20240503 08:18:24.830376 27997 catalog_manager.cc:1347] Generated new cluster ID: fc8b074ecc8b44ff9833aa82d4810935
I20240503 08:18:24.830437 27997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:24.831408 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:24.829867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:42971 (local address 127.0.0.1:39343)
0503 08:18:24.830033 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:24.830045 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:24.830067 (+ 22us) server_negotiation.cc:407] Connection header received
0503 08:18:24.830166 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:24.830172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:24.830200 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:24.830590 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:24.831032 (+ 442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:24.831034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:24.831035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:24.831038 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:24.831061 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:24.831068 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:24.831131 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:24.831233 (+ 102us) server_negotiation.cc:299] Negotiation successful
0503 08:18:24.831264 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20240503 08:18:24.832181 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:24.836773 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:24.833410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:50261 (local address 127.0.0.1:39343)
0503 08:18:24.833437 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:24.833439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:24.835796 (+ 2357us) server_negotiation.cc:407] Connection header received
0503 08:18:24.835861 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:24.835861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:24.835866 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:24.835920 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:24.836564 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:24.836565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:24.836565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:24.836569 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:24.836582 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:24.836586 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:24.836645 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:24.836702 (+ 57us) server_negotiation.cc:299] Negotiation successful
0503 08:18:24.836714 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12}
I20240503 08:18:24.844861 27997 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:24.845000 27997 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:24.848664 27997 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f91971a9fc0849f2a00aecc9b2d0bd9e: Generated new TSK 0
I20240503 08:18:24.848820 27997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:24.852981 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 27917
2024-05-03T08:18:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:24 dist-test-slave-3g67 krb5kdc[28016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:24 dist-test-slave-3g67 krb5kdc[28016](info): set up 2 sockets
May 03 08:18:24 dist-test-slave-3g67 krb5kdc[28016](info): commencing operation
krb5kdc: starting...
W20240503 08:18:25.808117 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:25 dist-test-slave-3g67 krb5kdc[28016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:18:25.939599 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:38745
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:38745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:26.016583 28029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:26.016784 28029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:26.016821 28029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:26.018087 28029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:26.018128 28029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:26.018148 28029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:26.018168 28029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:26.018185 28029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:26.019763 28029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:38745
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:38745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:26.019984 28029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:26.020174 28029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:26.024637 28037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:26.025204 28034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:26.025755 28035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:26.026360 28029 server_base.cc:1034] running on GCE node
I20240503 08:18:26.026556 28029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:26.026786 28029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:26.027920 28029 hybrid_clock.cc:648] HybridClock initialized: now 1714724306027899 us; error 32 us; skew 500 ppm
May 03 08:18:26 dist-test-slave-3g67 krb5kdc[28016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724306, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:26.030457 28029 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:18:26.030838 28029 webserver.cc:460] Webserver started at http://127.6.68.254:45991/ using document root <none> and password file <none>
I20240503 08:18:26.031009 28029 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:26.031051 28029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:26.031167 28029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:26.032200 28029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "64db51f7a58d42dcbed601d65e37d449"
format_stamp: "Formatted at 2024-05-03 08:18:26 on dist-test-slave-3g67"
server_key: "642af8b89d95f74aaf5d4600bdb4b6ad"
server_key_iv: "e383640ba38f4f493dc30151266b95a6"
server_key_version: "encryptionkey@0"
I20240503 08:18:26.032505 28029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "64db51f7a58d42dcbed601d65e37d449"
format_stamp: "Formatted at 2024-05-03 08:18:26 on dist-test-slave-3g67"
server_key: "642af8b89d95f74aaf5d4600bdb4b6ad"
server_key_iv: "e383640ba38f4f493dc30151266b95a6"
server_key_version: "encryptionkey@0"
I20240503 08:18:26.033701 28029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:26.034322 28044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:26.034478 28029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:26.034554 28029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "64db51f7a58d42dcbed601d65e37d449"
format_stamp: "Formatted at 2024-05-03 08:18:26 on dist-test-slave-3g67"
server_key: "642af8b89d95f74aaf5d4600bdb4b6ad"
server_key_iv: "e383640ba38f4f493dc30151266b95a6"
server_key_version: "encryptionkey@0"
I20240503 08:18:26.034615 28029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:26.049554 28029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:26.051393 28029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:26.051519 28029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:26.055166 28029 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:38745
I20240503 08:18:26.055205 28096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:38745 every 8 connection(s)
I20240503 08:18:26.055498 28029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:26.056242 28097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:26.058243 28097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449: Bootstrap starting.
I20240503 08:18:26.058734 28097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:26.058943 28097 log.cc:824] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:26.059540 28097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449: No bootstrap required, opened a new log
I20240503 08:18:26.060916 28097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64db51f7a58d42dcbed601d65e37d449" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38745 } }
I20240503 08:18:26.061029 28097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:26.061048 28097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64db51f7a58d42dcbed601d65e37d449, State: Initialized, Role: FOLLOWER
I20240503 08:18:26.061144 28097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64db51f7a58d42dcbed601d65e37d449" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38745 } }
I20240503 08:18:26.061215 28097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:26.061245 28097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:26.061290 28097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:26.061755 28097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64db51f7a58d42dcbed601d65e37d449" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38745 } }
I20240503 08:18:26.061834 28097 leader_election.cc:302] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64db51f7a58d42dcbed601d65e37d449; no voters:
I20240503 08:18:26.061959 28097 leader_election.cc:288] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:26.062153 28097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:26.062165 28100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:26.062327 28100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [term 1 LEADER]: Becoming Leader. State: Replica: 64db51f7a58d42dcbed601d65e37d449, State: Running, Role: LEADER
I20240503 08:18:26.062453 28100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64db51f7a58d42dcbed601d65e37d449" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38745 } }
I20240503 08:18:26.062870 28102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64db51f7a58d42dcbed601d65e37d449. Latest consensus state: current_term: 1 leader_uuid: "64db51f7a58d42dcbed601d65e37d449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64db51f7a58d42dcbed601d65e37d449" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38745 } } }
I20240503 08:18:26.062882 28101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64db51f7a58d42dcbed601d65e37d449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64db51f7a58d42dcbed601d65e37d449" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 38745 } } }
I20240503 08:18:26.062960 28102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:26.062990 28101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:26.063176 28109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:26.063696 28109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:26.063946 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 28029
I20240503 08:18:26.064044 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:26.064196 6419 external_mini_cluster.cc:1428] Setting key 4e00d292b7bfdd6085776c2a979e9c87
I20240503 08:18:26.065331 28109 catalog_manager.cc:1347] Generated new cluster ID: 74a94d5b0362440c9214fdfc60adc48e
I20240503 08:18:26.065387 28109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 03 08:18:26 dist-test-slave-3g67 krb5kdc[28016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:26.072831 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:26.064593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38377 (local address 127.6.68.254:38745)
0503 08:18:26.064728 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:26.064745 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:26.064927 (+ 182us) server_negotiation.cc:407] Connection header received
0503 08:18:26.065062 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:26.065068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:26.065112 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:26.065247 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:26.066032 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:26.066794 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:26.067374 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:26.067616 (+ 242us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:26.069747 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:26.069753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:26.069753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:26.069773 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:26.071422 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:26.071830 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:26.071832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:26.071832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:26.071875 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:26.072062 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:26.072062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:26.072063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:26.072241 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:26.072375 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:26.072532 (+ 157us) server_negotiation.cc:299] Negotiation successful
0503 08:18:26.072652 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
I20240503 08:18:26.073127 28109 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:26.073673 28109 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:26.074117 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:26.081926 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:26.075494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43367 (local address 127.6.68.254:38745)
0503 08:18:26.075529 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:26.075531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:26.075563 (+ 32us) server_negotiation.cc:407] Connection header received
0503 08:18:26.075638 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:26.075639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:26.075649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:26.075713 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:26.076420 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:26.077214 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:26.077935 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:26.078117 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:26.079115 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:26.079119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:26.079119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:26.079125 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:26.080745 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:26.081122 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:26.081123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:26.081124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:26.081146 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:26.081332 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:26.081333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:26.081333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:26.081429 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:26.081548 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:26.081656 (+ 108us) server_negotiation.cc:299] Negotiation successful
0503 08:18:26.081720 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:18:26.083552 28109 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 64db51f7a58d42dcbed601d65e37d449: Generated new TSK 0
I20240503 08:18:26.083760 28109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:26.086696 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 28029
W20240503 08:18:26.090564 28014 connection.cc:535] client connection to 127.6.68.254:38745 recv error: Network error: failed to read from TLS socket (remote: 127.6.68.254:38745): Connection reset by peer (error 104)
2024-05-03T08:18:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:26 dist-test-slave-3g67 krb5kdc[28128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:26 dist-test-slave-3g67 krb5kdc[28128](info): set up 2 sockets
May 03 08:18:26 dist-test-slave-3g67 krb5kdc[28128](info): commencing operation
krb5kdc: starting...
W20240503 08:18:27.036047 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:27 dist-test-slave-3g67 krb5kdc[28128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:18:27.151466 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:37419
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:27.231034 28141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:27.231223 28141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:27.231245 28141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:27.232524 28141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:27.232566 28141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:27.232586 28141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:27.232599 28141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:27.232627 28141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:27.234205 28141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:37419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:27.234395 28141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:27.234587 28141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:27.238965 28147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:27.239432 28149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:27.240175 28146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:27.240658 28141 server_base.cc:1034] running on GCE node
I20240503 08:18:27.240855 28141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:27.241047 28141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:27.242177 28141 hybrid_clock.cc:648] HybridClock initialized: now 1714724307242153 us; error 40 us; skew 500 ppm
May 03 08:18:27 dist-test-slave-3g67 krb5kdc[28128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:27.244429 28141 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:18:27.244942 28141 webserver.cc:460] Webserver started at http://127.6.68.254:39259/ using document root <none> and password file <none>
I20240503 08:18:27.245138 28141 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:27.245182 28141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:27.245294 28141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:27.246321 28141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "a205111c9ed444e4a8143b392dd024c0"
format_stamp: "Formatted at 2024-05-03 08:18:27 on dist-test-slave-3g67"
server_key: "19d481018e2d5957edd4b5f298585f5e"
server_key_iv: "07a408e525ea80f7f0d1214b6a9fa901"
server_key_version: "encryptionkey@0"
I20240503 08:18:27.246631 28141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "a205111c9ed444e4a8143b392dd024c0"
format_stamp: "Formatted at 2024-05-03 08:18:27 on dist-test-slave-3g67"
server_key: "19d481018e2d5957edd4b5f298585f5e"
server_key_iv: "07a408e525ea80f7f0d1214b6a9fa901"
server_key_version: "encryptionkey@0"
I20240503 08:18:27.247871 28141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:18:27.248554 28156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:27.248700 28141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:27.248782 28141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "a205111c9ed444e4a8143b392dd024c0"
format_stamp: "Formatted at 2024-05-03 08:18:27 on dist-test-slave-3g67"
server_key: "19d481018e2d5957edd4b5f298585f5e"
server_key_iv: "07a408e525ea80f7f0d1214b6a9fa901"
server_key_version: "encryptionkey@0"
I20240503 08:18:27.248847 28141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:27.259603 28141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:27.261165 28141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:27.261296 28141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:27.265159 28141 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:37419
I20240503 08:18:27.265183 28208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:37419 every 8 connection(s)
I20240503 08:18:27.265481 28141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:27.265825 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 28141
I20240503 08:18:27.266018 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:27.266222 6419 external_mini_cluster.cc:1428] Setting key 33feab2ba407737dc7fe9fd8b2727574
I20240503 08:18:27.266249 28209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:27.269479 28209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0: Bootstrap starting.
I20240503 08:18:27.270061 28209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:27.270335 28209 log.cc:824] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:27.270933 28209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0: No bootstrap required, opened a new log
I20240503 08:18:27.271169 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:27.266705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33273 (local address 127.6.68.254:37419)
0503 08:18:27.266859 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:27.266870 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:27.266896 (+ 26us) server_negotiation.cc:407] Connection header received
0503 08:18:27.267499 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:27.267507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:27.267561 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:27.267729 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:27.268089 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:27.269189 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:27.269821 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:27.270075 (+ 254us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:27.270553 (+ 478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:27.270561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:27.270561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:27.270746 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:27.270763 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:27.270769 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:27.270854 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:27.270976 (+ 122us) server_negotiation.cc:299] Negotiation successful
0503 08:18:27.271003 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20240503 08:18:27.272126 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:27.272823 28209 raft_consensus.cc:348] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a205111c9ed444e4a8143b392dd024c0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37419 } }
I20240503 08:18:27.272969 28209 raft_consensus.cc:374] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:27.273002 28209 raft_consensus.cc:729] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a205111c9ed444e4a8143b392dd024c0, State: Initialized, Role: FOLLOWER
I20240503 08:18:27.273110 28209 consensus_queue.cc:260] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a205111c9ed444e4a8143b392dd024c0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37419 } }
I20240503 08:18:27.273165 28209 raft_consensus.cc:388] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:27.273185 28209 raft_consensus.cc:482] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:27.273228 28209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:27.273921 28209 raft_consensus.cc:504] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a205111c9ed444e4a8143b392dd024c0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37419 } }
I20240503 08:18:27.274030 28209 leader_election.cc:302] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a205111c9ed444e4a8143b392dd024c0; no voters:
I20240503 08:18:27.274188 28209 leader_election.cc:288] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:27.274224 28219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:27.274369 28219 raft_consensus.cc:686] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [term 1 LEADER]: Becoming Leader. State: Replica: a205111c9ed444e4a8143b392dd024c0, State: Running, Role: LEADER
I20240503 08:18:27.274405 28209 sys_catalog.cc:564] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:27.274523 28219 consensus_queue.cc:237] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a205111c9ed444e4a8143b392dd024c0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37419 } }
I20240503 08:18:27.274873 28221 sys_catalog.cc:455] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a205111c9ed444e4a8143b392dd024c0. Latest consensus state: current_term: 1 leader_uuid: "a205111c9ed444e4a8143b392dd024c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a205111c9ed444e4a8143b392dd024c0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37419 } } }
I20240503 08:18:27.274953 28221 sys_catalog.cc:458] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:27.275141 28220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a205111c9ed444e4a8143b392dd024c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a205111c9ed444e4a8143b392dd024c0" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 37419 } } }
I20240503 08:18:27.275265 28220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:27.275426 28224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:27.275899 28224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:27.276438 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:27.273276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56065 (local address 127.6.68.254:37419)
0503 08:18:27.273308 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:27.273309 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:27.273531 (+ 222us) server_negotiation.cc:407] Connection header received
0503 08:18:27.273604 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:27.273605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:27.273612 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:27.273663 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:27.273986 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:27.274561 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:27.275581 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:27.275781 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:27.276101 (+ 320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:27.276104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:27.276104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:27.276109 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:27.276123 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:27.276127 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:27.276185 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:27.276349 (+ 164us) server_negotiation.cc:299] Negotiation successful
0503 08:18:27.276359 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240503 08:18:27.277386 28224 catalog_manager.cc:1347] Generated new cluster ID: d46a4a41ff394da186ca6ffc909fcc4d
I20240503 08:18:27.277436 28224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:27.286726 28224 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:27.287447 28224 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:27.297713 28224 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a205111c9ed444e4a8143b392dd024c0: Generated new TSK 0
I20240503 08:18:27.297881 28224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:27.304489 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 28141
W20240503 08:18:27.308952 28126 connection.cc:535] client connection to 127.6.68.254:37419 recv error: Network error: failed to read from TLS socket (remote: 127.6.68.254:37419): Connection reset by peer (error 104)
2024-05-03T08:18:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:27 dist-test-slave-3g67 krb5kdc[28240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:27 dist-test-slave-3g67 krb5kdc[28240](info): set up 2 sockets
May 03 08:18:27 dist-test-slave-3g67 krb5kdc[28240](info): commencing operation
krb5kdc: starting...
W20240503 08:18:28.264765 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:28 dist-test-slave-3g67 krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:18:28.379979 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45837
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:28.458539 28253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:28.458734 28253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:28.458771 28253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:28.460088 28253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:28.460132 28253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:28.460155 28253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:28.460175 28253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:28.460199 28253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:28.461835 28253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:45837
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:45837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:28.462054 28253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:28.462237 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:28.466544 28258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:28.467232 28259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:28.467767 28261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:28.468324 28253 server_base.cc:1034] running on GCE node
I20240503 08:18:28.468550 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:28.468873 28253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:28.470050 28253 hybrid_clock.cc:648] HybridClock initialized: now 1714724308470032 us; error 30 us; skew 500 ppm
May 03 08:18:28 dist-test-slave-3g67 krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724308, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:28.480834 28253 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:18:28.481297 28253 webserver.cc:460] Webserver started at http://127.6.68.254:42237/ using document root <none> and password file <none>
I20240503 08:18:28.481482 28253 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:28.481525 28253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:28.481635 28253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:28.482651 28253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "95bfea96377b4e39815e5723287376bb"
format_stamp: "Formatted at 2024-05-03 08:18:28 on dist-test-slave-3g67"
server_key: "667e910da2fb3113270787b308280b5e"
server_key_iv: "90f29373c08619f460340115e616415a"
server_key_version: "encryptionkey@0"
I20240503 08:18:28.482954 28253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "95bfea96377b4e39815e5723287376bb"
format_stamp: "Formatted at 2024-05-03 08:18:28 on dist-test-slave-3g67"
server_key: "667e910da2fb3113270787b308280b5e"
server_key_iv: "90f29373c08619f460340115e616415a"
server_key_version: "encryptionkey@0"
I20240503 08:18:28.484200 28253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:18:28.484898 28268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:28.485085 28253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:28.485163 28253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "95bfea96377b4e39815e5723287376bb"
format_stamp: "Formatted at 2024-05-03 08:18:28 on dist-test-slave-3g67"
server_key: "667e910da2fb3113270787b308280b5e"
server_key_iv: "90f29373c08619f460340115e616415a"
server_key_version: "encryptionkey@0"
I20240503 08:18:28.485227 28253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:28.488689 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:28.490708 28253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:28.490826 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:28.494446 28253 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:45837
I20240503 08:18:28.494489 28320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:45837 every 8 connection(s)
I20240503 08:18:28.494760 28253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:28.495811 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:28.497934 28321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb: Bootstrap starting.
I20240503 08:18:28.498431 28321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:28.498644 28321 log.cc:824] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:28.499251 28321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb: No bootstrap required, opened a new log
I20240503 08:18:28.500617 28321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfea96377b4e39815e5723287376bb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45837 } }
I20240503 08:18:28.500731 28321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:28.500747 28321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95bfea96377b4e39815e5723287376bb, State: Initialized, Role: FOLLOWER
I20240503 08:18:28.500844 28321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfea96377b4e39815e5723287376bb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45837 } }
I20240503 08:18:28.500913 28321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:28.500936 28321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:28.500960 28321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:28.501457 28321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfea96377b4e39815e5723287376bb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45837 } }
I20240503 08:18:28.501538 28321 leader_election.cc:302] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95bfea96377b4e39815e5723287376bb; no voters:
I20240503 08:18:28.501646 28321 leader_election.cc:288] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:28.501703 28324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:28.501850 28324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [term 1 LEADER]: Becoming Leader. State: Replica: 95bfea96377b4e39815e5723287376bb, State: Running, Role: LEADER
I20240503 08:18:28.501859 28321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:28.501951 28324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfea96377b4e39815e5723287376bb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45837 } }
I20240503 08:18:28.502285 28326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95bfea96377b4e39815e5723287376bb. Latest consensus state: current_term: 1 leader_uuid: "95bfea96377b4e39815e5723287376bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfea96377b4e39815e5723287376bb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45837 } } }
I20240503 08:18:28.502386 28325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95bfea96377b4e39815e5723287376bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bfea96377b4e39815e5723287376bb" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 45837 } } }
I20240503 08:18:28.502537 28325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:28.502426 28326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:28.502856 28333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:28.503293 28333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:28.504323 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 28253
I20240503 08:18:28.504441 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:28.504617 6419 external_mini_cluster.cc:1428] Setting key 4c54bb2788d11b390d2dad9922022174
I20240503 08:18:28.507997 28333 catalog_manager.cc:1347] Generated new cluster ID: d0271160d6af4522a5fb0d44e9ad2470
I20240503 08:18:28.508062 28333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:28.509320 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:28.505070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35069 (local address 127.6.68.254:45837)
0503 08:18:28.507992 (+ 2922us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:28.508001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:28.508018 (+ 17us) server_negotiation.cc:407] Connection header received
0503 08:18:28.508108 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:28.508114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:28.508142 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:28.508279 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:28.508976 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:28.508978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:28.508978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:28.508982 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:28.509002 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:28.509008 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:28.509076 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:28.509154 (+ 78us) server_negotiation.cc:299] Negotiation successful
0503 08:18:28.509189 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2891,"thread_start_us":70,"threads_started":1}
I20240503 08:18:28.512514 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:28.517127 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:28.515980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54425 (local address 127.6.68.254:45837)
0503 08:18:28.516019 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:28.516021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:28.516150 (+ 129us) server_negotiation.cc:407] Connection header received
0503 08:18:28.516218 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:28.516219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:28.516224 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:28.516275 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:28.516888 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:28.516890 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:28.516890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:28.516894 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:28.516909 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:28.516913 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:28.516970 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:28.517022 (+ 52us) server_negotiation.cc:299] Negotiation successful
0503 08:18:28.517038 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:18:28.529079 28333 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:28.529271 28333 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:28.537452 28333 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 95bfea96377b4e39815e5723287376bb: Generated new TSK 0
I20240503 08:18:28.537640 28333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:28.542289 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 28253
2024-05-03T08:18:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1237 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:28 dist-test-slave-3g67 krb5kdc[28352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:28 dist-test-slave-3g67 krb5kdc[28352](info): set up 2 sockets
May 03 08:18:28 dist-test-slave-3g67 krb5kdc[28352](info): commencing operation
krb5kdc: starting...
W20240503 08:18:29.496076 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:29 dist-test-slave-3g67 krb5kdc[28352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:18:29.615424 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:44091
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:42153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:44091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:29.693152 28365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:29.693346 28365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:29.693378 28365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:29.694628 28365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:29.694670 28365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:29.694692 28365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:29.694712 28365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:29.694730 28365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:29.696298 28365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:42153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:44091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:44091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:29.696513 28365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:29.696697 28365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:29.701056 28370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:29.701607 28373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:29.702185 28371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:29.702812 28365 server_base.cc:1034] running on GCE node
I20240503 08:18:29.702994 28365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:29.703184 28365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:29.704306 28365 hybrid_clock.cc:648] HybridClock initialized: now 1714724309704295 us; error 27 us; skew 500 ppm
May 03 08:18:29 dist-test-slave-3g67 krb5kdc[28352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724309, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:29.706550 28365 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:18:29.706928 28365 webserver.cc:460] Webserver started at http://127.6.68.254:45577/ using document root <none> and password file <none>
I20240503 08:18:29.707091 28365 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:29.707127 28365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:29.707203 28365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:29.708156 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "80e9cd12ac944c76ba8f9eb465e9c05f"
format_stamp: "Formatted at 2024-05-03 08:18:29 on dist-test-slave-3g67"
server_key: "6925f7bf8917ae6f2c4e2acc546ebd71"
server_key_iv: "5da0d468ff37546e84261e394cf8b5c7"
server_key_version: "encryptionkey@0"
I20240503 08:18:29.708433 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "80e9cd12ac944c76ba8f9eb465e9c05f"
format_stamp: "Formatted at 2024-05-03 08:18:29 on dist-test-slave-3g67"
server_key: "6925f7bf8917ae6f2c4e2acc546ebd71"
server_key_iv: "5da0d468ff37546e84261e394cf8b5c7"
server_key_version: "encryptionkey@0"
I20240503 08:18:29.709513 28365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:29.710132 28380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:29.710304 28365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:29.710373 28365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "80e9cd12ac944c76ba8f9eb465e9c05f"
format_stamp: "Formatted at 2024-05-03 08:18:29 on dist-test-slave-3g67"
server_key: "6925f7bf8917ae6f2c4e2acc546ebd71"
server_key_iv: "5da0d468ff37546e84261e394cf8b5c7"
server_key_version: "encryptionkey@0"
I20240503 08:18:29.710423 28365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:29.722258 28365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:29.723915 28365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:29.724010 28365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:29.727468 28365 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:44091
I20240503 08:18:29.727525 28432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:44091 every 8 connection(s)
I20240503 08:18:29.727825 28365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:29.728572 28433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:29.729990 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 28365
I20240503 08:18:29.730065 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:29.730223 6419 external_mini_cluster.cc:1428] Setting key 430fdd95a33d8445066400e67e44975b
I20240503 08:18:29.730712 28433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f: Bootstrap starting.
I20240503 08:18:29.731236 28433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:29.731514 28433 log.cc:824] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:29.732235 28433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f: No bootstrap required, opened a new log
I20240503 08:18:29.733983 28433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e9cd12ac944c76ba8f9eb465e9c05f" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44091 } }
I20240503 08:18:29.734138 28433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:29.734172 28433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e9cd12ac944c76ba8f9eb465e9c05f, State: Initialized, Role: FOLLOWER
I20240503 08:18:29.734324 28433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e9cd12ac944c76ba8f9eb465e9c05f" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44091 } }
I20240503 08:18:29.734406 28433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:29.734488 28433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:29.734544 28433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:29.735045 28433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e9cd12ac944c76ba8f9eb465e9c05f" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44091 } }
I20240503 08:18:29.735127 28433 leader_election.cc:302] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80e9cd12ac944c76ba8f9eb465e9c05f; no voters:
I20240503 08:18:29.735292 28433 leader_election.cc:288] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:29.735539 28433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [sys.catalog]: configured and running, proceeding with master startup.
May 03 08:18:29 dist-test-slave-3g67 krb5kdc[28352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:29.735337 28438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:29.736266 28438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [term 1 LEADER]: Becoming Leader. State: Replica: 80e9cd12ac944c76ba8f9eb465e9c05f, State: Running, Role: LEADER
I20240503 08:18:29.736449 28438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e9cd12ac944c76ba8f9eb465e9c05f" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44091 } }
I20240503 08:18:29.736754 28438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80e9cd12ac944c76ba8f9eb465e9c05f. Latest consensus state: current_term: 1 leader_uuid: "80e9cd12ac944c76ba8f9eb465e9c05f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e9cd12ac944c76ba8f9eb465e9c05f" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44091 } } }
I20240503 08:18:29.737227 28438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [sys.catalog]: This master's current role is: LEADER
W20240503 08:18:29.737416 28452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:18:29.737474 28452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:18:29.736922 28439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80e9cd12ac944c76ba8f9eb465e9c05f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e9cd12ac944c76ba8f9eb465e9c05f" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 44091 } } }
I20240503 08:18:29.737618 28439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:29.739744 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:29.730717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39021 (local address 127.6.68.254:44091)
0503 08:18:29.730857 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:29.730868 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:29.730889 (+ 21us) server_negotiation.cc:407] Connection header received
0503 08:18:29.731304 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:29.731311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:29.731363 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:29.731476 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:29.732421 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:29.733141 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:29.734000 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:29.734147 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:29.736583 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:29.736591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:29.736592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:29.736627 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:29.738513 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:29.738820 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:29.738824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:29.738824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:29.738866 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:29.739008 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:29.739009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:29.739009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:29.739185 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:29.739337 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:29.739434 (+ 97us) server_negotiation.cc:299] Negotiation successful
0503 08:18:29.739554 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20240503 08:18:29.740572 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:29.743764 28454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:29.744421 28454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:29.745865 28454 catalog_manager.cc:1347] Generated new cluster ID: 5a0e599639294e78988a8eb0ceb50ff2
I20240503 08:18:29.745918 28454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:29.747342 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:29.741809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36879 (local address 127.6.68.254:44091)
0503 08:18:29.741863 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:29.741865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:29.741875 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:18:29.741941 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:29.741942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:29.741951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:29.742014 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:29.742576 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:29.743296 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:29.743842 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:29.744036 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:29.744943 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:29.744946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:29.744946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:29.744952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:29.746434 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:29.746714 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:29.746715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:29.746716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:29.746737 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:29.746870 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:29.746870 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:29.746871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:29.746948 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:29.747049 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:29.747144 (+ 95us) server_negotiation.cc:299] Negotiation successful
0503 08:18:29.747219 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240503 08:18:29.760903 28454 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:29.761646 28454 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:29.770339 28454 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 80e9cd12ac944c76ba8f9eb465e9c05f: Generated new TSK 0
I20240503 08:18:29.770509 28454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:29.776310 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 28365
2024-05-03T08:18:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:29 dist-test-slave-3g67 krb5kdc[28464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:29 dist-test-slave-3g67 krb5kdc[28464](info): set up 2 sockets
May 03 08:18:29 dist-test-slave-3g67 krb5kdc[28464](info): commencing operation
krb5kdc: starting...
W20240503 08:18:30.732098 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:30 dist-test-slave-3g67 krb5kdc[28464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:18:30.843421 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:33121
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:33121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:30.919539 28477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:30.919744 28477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:30.919768 28477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:30.921013 28477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:30.921051 28477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:30.921064 28477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:30.921077 28477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:30.921087 28477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:30.922633 28477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:33121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:33121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:30.922789 28477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:30.922926 28477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:30.927215 28483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:30.927706 28482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:30.928344 28485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:30.928947 28477 server_base.cc:1034] running on GCE node
I20240503 08:18:30.929136 28477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:30.929343 28477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:30.930473 28477 hybrid_clock.cc:648] HybridClock initialized: now 1714724310930452 us; error 33 us; skew 500 ppm
May 03 08:18:30 dist-test-slave-3g67 krb5kdc[28464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724310, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:30.932807 28477 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:18:30.933177 28477 webserver.cc:460] Webserver started at http://127.6.68.254:37395/ using document root <none> and password file <none>
I20240503 08:18:30.933349 28477 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:30.933387 28477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:30.933482 28477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:30.934425 28477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "3f9c307923b44c5d8c9d051d5adc82e5"
format_stamp: "Formatted at 2024-05-03 08:18:30 on dist-test-slave-3g67"
server_key: "02737b06e96365f8f047c6b071ca849d"
server_key_iv: "3c26abccdfcf75a2d88e0e5c11922fda"
server_key_version: "encryptionkey@0"
I20240503 08:18:30.934720 28477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "3f9c307923b44c5d8c9d051d5adc82e5"
format_stamp: "Formatted at 2024-05-03 08:18:30 on dist-test-slave-3g67"
server_key: "02737b06e96365f8f047c6b071ca849d"
server_key_iv: "3c26abccdfcf75a2d88e0e5c11922fda"
server_key_version: "encryptionkey@0"
I20240503 08:18:30.935999 28477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:18:30.936610 28492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:30.936800 28477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:30.936868 28477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "3f9c307923b44c5d8c9d051d5adc82e5"
format_stamp: "Formatted at 2024-05-03 08:18:30 on dist-test-slave-3g67"
server_key: "02737b06e96365f8f047c6b071ca849d"
server_key_iv: "3c26abccdfcf75a2d88e0e5c11922fda"
server_key_version: "encryptionkey@0"
I20240503 08:18:30.936936 28477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:30.969087 28477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:30.971199 28477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:30.971318 28477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:30.975244 28477 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:33121
I20240503 08:18:30.975281 28544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:33121 every 8 connection(s)
I20240503 08:18:30.975566 28477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:30.976312 28545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:30.977432 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 28477
I20240503 08:18:30.977543 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:30.977728 6419 external_mini_cluster.cc:1428] Setting key 2859512cc3494fd2da6dec9a5be0aeb7
I20240503 08:18:30.978801 28545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5: Bootstrap starting.
I20240503 08:18:30.979292 28545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:30.979543 28545 log.cc:824] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:30.980154 28545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5: No bootstrap required, opened a new log
I20240503 08:18:30.981843 28545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c307923b44c5d8c9d051d5adc82e5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33121 } }
I20240503 08:18:30.981979 28545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:30.982056 28545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9c307923b44c5d8c9d051d5adc82e5, State: Initialized, Role: FOLLOWER
I20240503 08:18:30.982098 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:30.978253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56795 (local address 127.6.68.254:33121)
0503 08:18:30.978429 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:30.978438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:30.978455 (+ 17us) server_negotiation.cc:407] Connection header received
0503 08:18:30.978880 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:30.978888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:30.978939 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:30.979064 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:30.979454 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:30.980300 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:30.980860 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:30.981096 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:30.981560 (+ 464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:30.981567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:30.981568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:30.981704 (+ 136us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:30.981725 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:30.981731 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:30.981795 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:30.981871 (+ 76us) server_negotiation.cc:299] Negotiation successful
0503 08:18:30.981908 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20240503 08:18:30.982182 28545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c307923b44c5d8c9d051d5adc82e5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33121 } }
I20240503 08:18:30.982254 28545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:30.982286 28545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:30.982334 28545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:30.982944 28545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c307923b44c5d8c9d051d5adc82e5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33121 } }
I20240503 08:18:30.983063 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:30.983055 28545 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f9c307923b44c5d8c9d051d5adc82e5; no voters:
I20240503 08:18:30.983215 28545 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:30.983266 28550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:30.983410 28545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:30.983440 28550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [term 1 LEADER]: Becoming Leader. State: Replica: 3f9c307923b44c5d8c9d051d5adc82e5, State: Running, Role: LEADER
I20240503 08:18:30.983534 28550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c307923b44c5d8c9d051d5adc82e5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33121 } }
I20240503 08:18:30.983850 28551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f9c307923b44c5d8c9d051d5adc82e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c307923b44c5d8c9d051d5adc82e5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33121 } } }
I20240503 08:18:30.983968 28551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:30.984150 28551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9c307923b44c5d8c9d051d5adc82e5. Latest consensus state: current_term: 1 leader_uuid: "3f9c307923b44c5d8c9d051d5adc82e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9c307923b44c5d8c9d051d5adc82e5" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33121 } } }
I20240503 08:18:30.984251 28551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:30.984575 28561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:30.984997 28561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:30.986728 28561 catalog_manager.cc:1347] Generated new cluster ID: 9e5511c5c4b747db9a3d8cfb5dd65d9d
I20240503 08:18:30.986774 28561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:30.987526 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:30.984546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50001 (local address 127.6.68.254:33121)
0503 08:18:30.984581 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:30.984583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:30.984792 (+ 209us) server_negotiation.cc:407] Connection header received
0503 08:18:30.984861 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:30.984861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:30.984871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:30.984925 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:30.985260 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:30.985880 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:30.986588 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:30.986761 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:30.987280 (+ 519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:30.987283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:30.987283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:30.987287 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:30.987302 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:30.987305 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:30.987361 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:30.987432 (+ 71us) server_negotiation.cc:299] Negotiation successful
0503 08:18:30.987458 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240503 08:18:30.994777 28561 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:30.995296 28561 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:31.003017 28561 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3f9c307923b44c5d8c9d051d5adc82e5: Generated new TSK 0
I20240503 08:18:31.003182 28561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:31.006829 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 28477
2024-05-03T08:18:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1228 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:31 dist-test-slave-3g67 krb5kdc[28576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:31 dist-test-slave-3g67 krb5kdc[28576](info): set up 2 sockets
May 03 08:18:31 dist-test-slave-3g67 krb5kdc[28576](info): commencing operation
krb5kdc: starting...
W20240503 08:18:31.960731 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:32 dist-test-slave-3g67 krb5kdc[28576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:18:32.072654 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:33319
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:33319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:32.150031 28589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:32.150238 28589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:32.150287 28589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:32.151566 28589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:32.151610 28589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:32.151656 28589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:32.151674 28589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:32.151710 28589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:32.153298 28589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:33319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:33319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:32.153519 28589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:32.153702 28589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:32.158059 28594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:32.158478 28595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:32.159143 28597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:32.159708 28589 server_base.cc:1034] running on GCE node
I20240503 08:18:32.159904 28589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:32.160106 28589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:32.161247 28589 hybrid_clock.cc:648] HybridClock initialized: now 1714724312161235 us; error 29 us; skew 500 ppm
May 03 08:18:32 dist-test-slave-3g67 krb5kdc[28576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724312, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:32.163512 28589 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:18:32.163937 28589 webserver.cc:460] Webserver started at http://127.6.68.254:42105/ using document root <none> and password file <none>
I20240503 08:18:32.164108 28589 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:32.164142 28589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:32.164224 28589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:32.165166 28589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "37bbfeb9c694478a99f8ee7594c13c00"
format_stamp: "Formatted at 2024-05-03 08:18:32 on dist-test-slave-3g67"
server_key: "99c707f2c740f6740d934e08ac235981"
server_key_iv: "e1fefebcee2ae090438eb13069f76f91"
server_key_version: "encryptionkey@0"
I20240503 08:18:32.165422 28589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "37bbfeb9c694478a99f8ee7594c13c00"
format_stamp: "Formatted at 2024-05-03 08:18:32 on dist-test-slave-3g67"
server_key: "99c707f2c740f6740d934e08ac235981"
server_key_iv: "e1fefebcee2ae090438eb13069f76f91"
server_key_version: "encryptionkey@0"
I20240503 08:18:32.166513 28589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:18:32.167106 28604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:32.167264 28589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:18:32.167335 28589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "37bbfeb9c694478a99f8ee7594c13c00"
format_stamp: "Formatted at 2024-05-03 08:18:32 on dist-test-slave-3g67"
server_key: "99c707f2c740f6740d934e08ac235981"
server_key_iv: "e1fefebcee2ae090438eb13069f76f91"
server_key_version: "encryptionkey@0"
I20240503 08:18:32.167402 28589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:32.170795 28589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:32.172564 28589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:32.172665 28589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:32.176231 28589 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:33319
I20240503 08:18:32.176290 28656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:33319 every 8 connection(s)
I20240503 08:18:32.176556 28589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:32.177347 28657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:32.179311 28657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00: Bootstrap starting.
I20240503 08:18:32.179831 28657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:32.180038 28657 log.cc:824] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:32.180622 28657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00: No bootstrap required, opened a new log
I20240503 08:18:32.181962 28657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bbfeb9c694478a99f8ee7594c13c00" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33319 } }
I20240503 08:18:32.182077 28657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:18:32.182093 28657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bbfeb9c694478a99f8ee7594c13c00, State: Initialized, Role: FOLLOWER
I20240503 08:18:32.182210 28657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bbfeb9c694478a99f8ee7594c13c00" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33319 } }
I20240503 08:18:32.182274 28657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:18:32.182296 28657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:18:32.182348 28657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:32.182853 28657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bbfeb9c694478a99f8ee7594c13c00" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33319 } }
I20240503 08:18:32.182933 28657 leader_election.cc:302] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37bbfeb9c694478a99f8ee7594c13c00; no voters:
I20240503 08:18:32.183077 28657 leader_election.cc:288] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:18:32.183151 28660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:32.183298 28657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:32.183305 28660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [term 1 LEADER]: Becoming Leader. State: Replica: 37bbfeb9c694478a99f8ee7594c13c00, State: Running, Role: LEADER
I20240503 08:18:32.183485 28660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bbfeb9c694478a99f8ee7594c13c00" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33319 } }
I20240503 08:18:32.183866 28662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37bbfeb9c694478a99f8ee7594c13c00. Latest consensus state: current_term: 1 leader_uuid: "37bbfeb9c694478a99f8ee7594c13c00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bbfeb9c694478a99f8ee7594c13c00" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33319 } } }
I20240503 08:18:32.183984 28661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37bbfeb9c694478a99f8ee7594c13c00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bbfeb9c694478a99f8ee7594c13c00" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 33319 } } }
I20240503 08:18:32.184013 28662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:32.184113 28661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:32.184448 28669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:32.184873 28669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:32.186156 28669 catalog_manager.cc:1347] Generated new cluster ID: 66569dbae2bb42a6b79e53d5d18db6f7
I20240503 08:18:32.186208 28669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:32.186494 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 28589
I20240503 08:18:32.186590 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:32.186774 6419 external_mini_cluster.cc:1428] Setting key b3ed2dd8ed6adc5e27b96422860973ab
I20240503 08:18:32.193311 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:32.187228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36865 (local address 127.6.68.254:33319)
0503 08:18:32.187395 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:32.187406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:32.191894 (+ 4488us) server_negotiation.cc:407] Connection header received
0503 08:18:32.192063 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:32.192069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:32.192100 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:32.192244 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:32.192946 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:32.192949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:32.192950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:32.192953 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:32.192970 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:32.192977 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:32.193049 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:32.193116 (+ 67us) server_negotiation.cc:299] Negotiation successful
0503 08:18:32.193149 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20240503 08:18:32.194236 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:32.200886 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:32.195741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57965 (local address 127.6.68.254:33319)
0503 08:18:32.195774 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:32.195776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:32.199856 (+ 4080us) server_negotiation.cc:407] Connection header received
0503 08:18:32.199929 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:32.199930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:32.199935 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:32.199990 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:32.200658 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:32.200660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:32.200660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0503 08:18:32.200664 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:32.200679 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0503 08:18:32.200684 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:32.200737 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:32.200798 (+ 61us) server_negotiation.cc:299] Negotiation successful
0503 08:18:32.200811 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240503 08:18:32.202729 28669 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:32.202859 28669 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:32.208570 28669 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 37bbfeb9c694478a99f8ee7594c13c00: Generated new TSK 0
I20240503 08:18:32.208745 28669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:32.217008 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 28589
2024-05-03T08:18:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1212 ms)
[----------] 18 tests from AuthTokenIssuingTest (22105 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:32 dist-test-slave-3g67 krb5kdc[28688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:32 dist-test-slave-3g67 krb5kdc[28688](info): set up 2 sockets
May 03 08:18:32 dist-test-slave-3g67 krb5kdc[28688](info): commencing operation
krb5kdc: starting...
W20240503 08:18:33.176230 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:18:33.294520 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40951
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40951,127.6.68.253:41951,127.6.68.252:43125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:33.373615 28701 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240503 08:18:33.373837 28701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:33.373863 28701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:33.373895 28701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:33.375206 28701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:33.375248 28701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:33.375270 28701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:33.375286 28701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:33.375303 28701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:33.376825 28701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40951,127.6.68.253:41951,127.6.68.252:43125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:33.377046 28701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:33.377238 28701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:33.381897 28706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:33.382407 28709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:33.382930 28707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:33.383647 28701 server_base.cc:1034] running on GCE node
I20240503 08:18:33.383880 28701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:33.384112 28701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:33.385242 28701 hybrid_clock.cc:648] HybridClock initialized: now 1714724313385222 us; error 37 us; skew 500 ppm
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:33.387562 28701 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:18:33.387984 28701 webserver.cc:460] Webserver started at http://127.6.68.254:35381/ using document root <none> and password file <none>
I20240503 08:18:33.388157 28701 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:33.388200 28701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:33.388303 28701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:33.389297 28701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "25ce589605f1406b9bf13c74434c307b"
format_stamp: "Formatted at 2024-05-03 08:18:33 on dist-test-slave-3g67"
server_key: "00eb2b75872139e796d48fa544464819"
server_key_iv: "8b97a916f0421881222d0f769187b51a"
server_key_version: "encryptionkey@0"
I20240503 08:18:33.389595 28701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "25ce589605f1406b9bf13c74434c307b"
format_stamp: "Formatted at 2024-05-03 08:18:33 on dist-test-slave-3g67"
server_key: "00eb2b75872139e796d48fa544464819"
server_key_iv: "8b97a916f0421881222d0f769187b51a"
server_key_version: "encryptionkey@0"
I20240503 08:18:33.390820 28701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240503 08:18:33.391503 28716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:33.391713 28701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:33.391793 28701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "25ce589605f1406b9bf13c74434c307b"
format_stamp: "Formatted at 2024-05-03 08:18:33 on dist-test-slave-3g67"
server_key: "00eb2b75872139e796d48fa544464819"
server_key_iv: "8b97a916f0421881222d0f769187b51a"
server_key_version: "encryptionkey@0"
I20240503 08:18:33.391856 28701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:33.410709 28701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:33.412922 28701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:33.413055 28701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:33.416745 28701 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:40951
I20240503 08:18:33.416790 28768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:40951 every 8 connection(s)
I20240503 08:18:33.417088 28701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:33.417950 28769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:33.418947 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 28701
I20240503 08:18:33.419037 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:33.419229 6419 external_mini_cluster.cc:1428] Setting key 2ac1015fad0b13cdbcfea58f6e6c6233
I20240503 08:18:33.419371 28769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.68.253@KRBTEST.COM; defaulting to no policy
I20240503 08:18:33.428450 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.420027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40951 (local address 127.0.0.1:49790)
0503 08:18:33.420184 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.420218 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:33.420873 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:33.421161 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:33.421163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:33.421496 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:33.421751 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.421755 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.423092 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.423093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:33.423440 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.423441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.423577 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.424163 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:33.424178 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:33.425499 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:33.427314 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.427320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.427323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.427498 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.427723 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.427724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.427724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.427769 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.428025 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:33.428028 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:33.428080 (+ 52us) client_negotiation.cc:769] Sending connection context
0503 08:18:33.428162 (+ 82us) client_negotiation.cc:240] Negotiation successful
0503 08:18:33.428330 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":67,"thread_start_us":75,"threads_started":1}
I20240503 08:18:33.428454 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.420146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.6.68.254:40951)
0503 08:18:33.420301 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.420309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.420339 (+ 30us) server_negotiation.cc:407] Connection header received
0503 08:18:33.421027 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.421028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.421045 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.421117 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.421848 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.422984 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.423542 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.423819 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.425593 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.425602 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.425602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.425610 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.427233 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.427580 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.427582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.427582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.427642 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.427817 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.427818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.427818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.427973 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.428033 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.428178 (+ 145us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.428262 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.253@KRBTEST.COM" created.
I20240503 08:18:33.429211 28769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:18:33.429570 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.429442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41951 (local address 127.0.0.1:56058)
0503 08:18:33.429494 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.429540 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20240503 08:18:33.429716 28717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41951: connect: Connection refused (error 111)
I20240503 08:18:33.430105 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.430020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41951 (local address 127.0.0.1:56066)
0503 08:18:33.430064 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.430088 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240503 08:18:33.430194 28769 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.253:41951: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41951: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab.
Entry for principal kudu/127.6.68.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab.
I20240503 08:18:33.445945 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.253:41951
--webserver_interface=127.6.68.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab
--principal=kudu/127.6.68.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40951,127.6.68.253:41951,127.6.68.252:43125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:18:33.479403 28769 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } attempt: 1
I20240503 08:18:33.480032 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.479881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41951 (local address 127.0.0.1:56076)
0503 08:18:33.479952 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.479993 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20240503 08:18:33.480417 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.480325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41951 (local address 127.0.0.1:56080)
0503 08:18:33.480370 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.480396 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240503 08:18:33.480522 28769 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.253:41951: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41951: connect: Connection refused (error 111)
W20240503 08:18:33.523797 28776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240503 08:18:33.523989 28776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:33.524006 28776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:33.524025 28776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:33.525315 28776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:33.525360 28776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:33.525373 28776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:33.525385 28776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:33.525396 28776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:33.526836 28776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40951,127.6.68.253:41951,127.6.68.252:43125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab
--principal=kudu/127.6.68.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.253:41951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.68.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:33.527034 28776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:33.527226 28776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:33.531656 28781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:33.532224 28784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:33.532714 28782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:33.533458 28776 server_base.cc:1034] running on GCE node
I20240503 08:18:33.533639 28776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:33.533881 28776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:33.535038 28776 hybrid_clock.cc:648] HybridClock initialized: now 1714724313535023 us; error 32 us; skew 500 ppm
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:33.542683 28776 init.cc:377] Logged in from keytab as kudu/127.6.68.253@KRBTEST.COM (short username kudu)
I20240503 08:18:33.543121 28776 webserver.cc:460] Webserver started at http://127.6.68.253:39489/ using document root <none> and password file <none>
I20240503 08:18:33.543303 28776 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:33.543350 28776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:33.543458 28776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:33.544498 28776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/data/instance:
uuid: "0b43814cd867460990000f0a8300fb1d"
format_stamp: "Formatted at 2024-05-03 08:18:33 on dist-test-slave-3g67"
server_key: "42a7224533a2258d4b7e8804beb8f771"
server_key_iv: "9a52a02ebe56fe155034281052d94078"
server_key_version: "encryptionkey@0"
I20240503 08:18:33.544819 28776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/wal/instance:
uuid: "0b43814cd867460990000f0a8300fb1d"
format_stamp: "Formatted at 2024-05-03 08:18:33 on dist-test-slave-3g67"
server_key: "42a7224533a2258d4b7e8804beb8f771"
server_key_iv: "9a52a02ebe56fe155034281052d94078"
server_key_version: "encryptionkey@0"
I20240503 08:18:33.545969 28776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:18:33.546591 28791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:33.546764 28776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:33.546841 28776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/wal
uuid: "0b43814cd867460990000f0a8300fb1d"
format_stamp: "Formatted at 2024-05-03 08:18:33 on dist-test-slave-3g67"
server_key: "42a7224533a2258d4b7e8804beb8f771"
server_key_iv: "9a52a02ebe56fe155034281052d94078"
server_key_version: "encryptionkey@0"
I20240503 08:18:33.546902 28776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:33.548676 28769 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } attempt: 2
I20240503 08:18:33.549088 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.548961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41951 (local address 127.0.0.1:56086)
0503 08:18:33.549021 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.549057 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20240503 08:18:33.549428 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.549355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41951 (local address 127.0.0.1:56102)
0503 08:18:33.549392 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.549412 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20240503 08:18:33.549525 28769 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.253:41951: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41951: connect: Connection refused (error 111)
I20240503 08:18:33.556478 28776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:33.558482 28776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:33.558584 28776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:33.562335 28776 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.253:41951
I20240503 08:18:33.562363 28843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.253:41951 every 8 connection(s)
I20240503 08:18:33.562685 28776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
I20240503 08:18:33.563491 28844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:33.564913 28844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:33.570251 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 28776
I20240503 08:18:33.570364 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-1/wal/instance
I20240503 08:18:33.570524 6419 external_mini_cluster.cc:1428] Setting key 688d086f19880fa76154a22e9492dd5b
I20240503 08:18:33.572698 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.565542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40951 (local address 127.0.0.1:49792)
0503 08:18:33.565690 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.565719 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:33.566144 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:33.566355 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:33.566358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:33.566642 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:33.566923 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.566926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.567523 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.567524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:33.567930 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.567932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.568031 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.568551 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:33.568563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:33.569858 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:33.571487 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.571493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.571496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.571822 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.571997 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.571998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.571998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.572046 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.572286 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:33.572288 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:33.572387 (+ 99us) client_negotiation.cc:769] Sending connection context
0503 08:18:33.572466 (+ 79us) client_negotiation.cc:240] Negotiation successful
0503 08:18:33.572561 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20240503 08:18:33.572747 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.565583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.6.68.254:40951)
0503 08:18:33.565652 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.565654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.565816 (+ 162us) server_negotiation.cc:407] Connection header received
0503 08:18:33.566234 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.566235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.566245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.566301 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.567008 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.567455 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.567996 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.568125 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.569946 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.569951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.569951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.569958 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.571402 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.571900 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.571902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.571902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.571931 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.572101 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.572102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.572102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.572229 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.572297 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.572500 (+ 203us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.572595 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240503 08:18:33.573361 28844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.68.252@KRBTEST.COM; defaulting to no policy
I20240503 08:18:33.579923 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.573626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41951 (local address 127.0.0.1:56108)
0503 08:18:33.573670 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.573681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:33.573719 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:33.573966 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:33.573968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:33.574132 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:33.574341 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.574343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.575127 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.575128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:33.575710 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.575712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.575793 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.576220 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:33.576225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:33.577467 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:33.579208 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.579209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.579209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.579337 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.579471 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.579471 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.579472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.579490 (+ 18us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.579691 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:33.579691 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:33.579736 (+ 45us) client_negotiation.cc:769] Sending connection context
0503 08:18:33.579786 (+ 50us) client_negotiation.cc:240] Negotiation successful
0503 08:18:33.579854 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20240503 08:18:33.580040 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.573688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.6.68.253:41951)
0503 08:18:33.573838 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.573845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.573858 (+ 13us) server_negotiation.cc:407] Connection header received
0503 08:18:33.573877 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.573878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.573887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.573941 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.574419 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.575041 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.575786 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.575912 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.577570 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.577580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.577580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.577586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.579126 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.579399 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.579401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.579402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.579424 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.579521 (+ 97us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.579522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.579522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.579614 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.579724 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.579822 (+ 98us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.579912 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":128,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.252@KRBTEST.COM" created.
I20240503 08:18:33.580528 28844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:18:33.580842 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.580742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35162)
0503 08:18:33.580786 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.580826 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240503 08:18:33.580951 28793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
I20240503 08:18:33.581374 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.581265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35176)
0503 08:18:33.581315 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.581342 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20240503 08:18:33.581483 28844 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:43125: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab.
Entry for principal kudu/127.6.68.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab.
I20240503 08:18:33.594902 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.252:43125
--webserver_interface=127.6.68.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab
--principal=kudu/127.6.68.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40951,127.6.68.253:41951,127.6.68.252:43125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:18:33.630760 28844 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } attempt: 1
I20240503 08:18:33.631320 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.631155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35186)
0503 08:18:33.631246 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.631289 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20240503 08:18:33.631701 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.631585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35194)
0503 08:18:33.631657 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.631682 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20240503 08:18:33.631806 28844 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:43125: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
I20240503 08:18:33.640672 28769 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } attempt: 3
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
I20240503 08:18:33.647614 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.640999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41951 (local address 127.0.0.1:56110)
0503 08:18:33.641053 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.641067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:33.641130 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:33.641264 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:33.641265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:33.641500 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:33.641734 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.641736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.642388 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.642389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:33.642947 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.642949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.643020 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.643499 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:33.643503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:33.645307 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:33.646916 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.646917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.646917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.647041 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.647174 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.647174 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.647174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.647193 (+ 19us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.647379 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:33.647380 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:33.647414 (+ 34us) client_negotiation.cc:769] Sending connection context
0503 08:18:33.647457 (+ 43us) client_negotiation.cc:240] Negotiation successful
0503 08:18:33.647528 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20240503 08:18:33.647697 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.641098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.6.68.253:41951)
0503 08:18:33.641149 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.641151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.641160 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:33.641184 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.641185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.641194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.641245 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.641814 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.642313 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.643024 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.643134 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.645393 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.645397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.645398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.645405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.646832 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.647099 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.647100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.647100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.647121 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.647225 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.647225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.647226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.647329 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.647389 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.647478 (+ 89us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.647580 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240503 08:18:33.648311 28769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:18:33.648630 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.648517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35200)
0503 08:18:33.648566 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.648600 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20240503 08:18:33.648926 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.648851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35216)
0503 08:18:33.648886 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.648908 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240503 08:18:33.649017 28769 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:43125: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
W20240503 08:18:33.675639 28851 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240503 08:18:33.675877 28851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:33.675894 28851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:33.675913 28851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:33.677183 28851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:33.677225 28851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:33.677238 28851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:33.677250 28851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:33.677260 28851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:33.678794 28851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40951,127.6.68.253:41951,127.6.68.252:43125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab
--principal=kudu/127.6.68.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.252:43125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.68.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:33.678999 28851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:33.679193 28851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:18:33.680184 28769 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } attempt: 1
I20240503 08:18:33.680652 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.680499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35218)
0503 08:18:33.680569 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.680616 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20240503 08:18:33.681035 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.680944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35224)
0503 08:18:33.680989 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.681014 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240503 08:18:33.681134 28769 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:43125: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
W20240503 08:18:33.683959 28857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:33.684273 28859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:33.685022 28856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:33.685613 28851 server_base.cc:1034] running on GCE node
I20240503 08:18:33.685851 28851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:33.686075 28851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:33.687230 28851 hybrid_clock.cc:648] HybridClock initialized: now 1714724313687211 us; error 35 us; skew 500 ppm
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:33.689806 28851 init.cc:377] Logged in from keytab as kudu/127.6.68.252@KRBTEST.COM (short username kudu)
I20240503 08:18:33.690240 28851 webserver.cc:460] Webserver started at http://127.6.68.252:42587/ using document root <none> and password file <none>
I20240503 08:18:33.690423 28851 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:33.690462 28851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:33.690570 28851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:33.691562 28851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/data/instance:
uuid: "c2c3a884b80840348d22d8237d527759"
format_stamp: "Formatted at 2024-05-03 08:18:33 on dist-test-slave-3g67"
server_key: "846ba6139372d20df0c0f5ad96a2d373"
server_key_iv: "28541d90fb756dec8a9d75a3afc33266"
server_key_version: "encryptionkey@0"
I20240503 08:18:33.691926 28851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/wal/instance:
uuid: "c2c3a884b80840348d22d8237d527759"
format_stamp: "Formatted at 2024-05-03 08:18:33 on dist-test-slave-3g67"
server_key: "846ba6139372d20df0c0f5ad96a2d373"
server_key_iv: "28541d90fb756dec8a9d75a3afc33266"
server_key_version: "encryptionkey@0"
I20240503 08:18:33.693164 28851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:33.693879 28866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:33.694041 28851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:33.694120 28851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/wal
uuid: "c2c3a884b80840348d22d8237d527759"
format_stamp: "Formatted at 2024-05-03 08:18:33 on dist-test-slave-3g67"
server_key: "846ba6139372d20df0c0f5ad96a2d373"
server_key_iv: "28541d90fb756dec8a9d75a3afc33266"
server_key_version: "encryptionkey@0"
I20240503 08:18:33.694202 28851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:33.699934 28844 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } attempt: 2
I20240503 08:18:33.700425 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.700256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35238)
0503 08:18:33.700335 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.700389 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20240503 08:18:33.700754 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.700678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35252)
0503 08:18:33.700717 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.700738 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20240503 08:18:33.700831 28844 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:43125: Network error: Client connection negotiation failed: client connection to 127.6.68.252:43125: connect: Connection refused (error 111)
I20240503 08:18:33.705619 28851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:33.707254 28851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:33.707355 28851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:33.711133 28851 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.252:43125
I20240503 08:18:33.711162 28918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.252:43125 every 8 connection(s)
I20240503 08:18:33.711468 28851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
I20240503 08:18:33.712330 28919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:33.713726 28919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:33.719421 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 28851
I20240503 08:18:33.719545 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714724013808011-6419-0/minicluster-data/master-2/wal/instance
I20240503 08:18:33.719769 6419 external_mini_cluster.cc:1428] Setting key ae418c39b958f827daeadf87bc88f959
I20240503 08:18:33.721450 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.714396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.6.68.254:40951)
0503 08:18:33.714462 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.714464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.714612 (+ 148us) server_negotiation.cc:407] Connection header received
0503 08:18:33.715052 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.715053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.715063 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.715115 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.715873 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.716318 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.716801 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.716904 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.718764 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.718769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.718769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.718777 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.720299 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.720660 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.720661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.720662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.720686 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.720863 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.720864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.720864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.720962 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.721020 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.721253 (+ 233us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.721361 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240503 08:18:33.721571 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.714341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40951 (local address 127.0.0.1:49806)
0503 08:18:33.714504 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.714532 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:33.714971 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:33.715171 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:33.715174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:33.715473 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:33.715801 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.715805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.716388 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.716389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:33.716745 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.716746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.716845 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.717350 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:33.717362 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:33.718660 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:33.720378 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.720384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.720388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.720591 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.720749 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.720750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.720750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.720808 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.721016 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:33.721020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:33.721134 (+ 114us) client_negotiation.cc:769] Sending connection context
0503 08:18:33.721235 (+ 101us) client_negotiation.cc:240] Negotiation successful
0503 08:18:33.721376 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20240503 08:18:33.722105 28919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
I20240503 08:18:33.728057 28923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.720151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.6.68.252:43125)
0503 08:18:33.720303 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.720311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.720390 (+ 79us) server_negotiation.cc:407] Connection header received
0503 08:18:33.720460 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.720462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.720477 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.720537 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.721239 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.722058 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.722714 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.722858 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.725034 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.725041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.725042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.725048 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.726867 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.727274 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.727276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.727277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.727300 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.727465 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.727466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.727467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.727598 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.727796 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.727883 (+ 87us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.727944 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20240503 08:18:33.728263 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.722435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.6.68.253:41951)
0503 08:18:33.722494 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.722495 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.722503 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:18:33.722524 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.722525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.722532 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.722578 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.723323 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.723763 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.724343 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.724513 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.725873 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.725878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.725878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.725884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.727320 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.727610 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.727611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.727611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.727655 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.727821 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.727822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.727822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.727922 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.727988 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.728121 (+ 133us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.728193 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240503 08:18:33.728260 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.722359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41951 (local address 127.0.0.1:56120)
0503 08:18:33.722408 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.722418 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:33.722466 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:33.722591 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:33.722592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:33.722746 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:33.723238 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.723241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.723832 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.723833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:33.724269 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.724271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.724326 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.724739 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:33.724743 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:33.725792 (+ 1049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:33.727385 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.727385 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.727386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.727541 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.727737 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.727738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.727738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.727764 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.727984 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:33.727985 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:33.728021 (+ 36us) client_negotiation.cc:769] Sending connection context
0503 08:18:33.728095 (+ 74us) client_negotiation.cc:240] Negotiation successful
0503 08:18:33.728188 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20240503 08:18:33.728935 28919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:33.734874 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.729173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35268)
0503 08:18:33.729236 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.729247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:33.729287 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:33.729458 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:33.729459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:33.729578 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:33.729685 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.729686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.730382 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.730383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:33.730797 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.730799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.730849 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.731249 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:33.731252 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:33.732435 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:33.734089 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.734089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.734090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.734223 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.734370 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.734370 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.734371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.734392 (+ 21us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.734589 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:33.734589 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:33.734619 (+ 30us) client_negotiation.cc:769] Sending connection context
0503 08:18:33.734676 (+ 57us) client_negotiation.cc:240] Negotiation successful
0503 08:18:33.734761 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20240503 08:18:33.734966 28923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.729247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.6.68.252:43125)
0503 08:18:33.729368 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.729369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.729375 (+ 6us) server_negotiation.cc:407] Connection header received
0503 08:18:33.729389 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.729389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.729396 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.729446 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.729756 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.730319 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.730861 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.731019 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.732514 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.732518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.732518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.732524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.734013 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.734287 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.734288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.734288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.734310 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.734432 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.734433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.734433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.734537 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.734599 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.734737 (+ 138us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.734822 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":108}
I20240503 08:18:33.735883 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.729212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.6.68.254:40951)
0503 08:18:33.729255 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.729257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.729300 (+ 43us) server_negotiation.cc:407] Connection header received
0503 08:18:33.729366 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.729367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.729375 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.729419 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.730083 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.730550 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.731120 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.731328 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.733205 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.733209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.733210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.733223 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.734603 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.734872 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.734873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.734873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.734894 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.735388 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.735389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.735389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.735485 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.735586 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.735755 (+ 169us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.735821 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:18:33.736938 28919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759: Bootstrap starting.
I20240503 08:18:33.737598 28919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:33.737865 28919 log.cc:824] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:33.738653 28919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759: No bootstrap required, opened a new log
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
I20240503 08:18:33.740520 28919 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } }
I20240503 08:18:33.740634 28919 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c3a884b80840348d22d8237d527759, State: Initialized, Role: FOLLOWER
I20240503 08:18:33.740696 28919 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } }
I20240503 08:18:33.741004 28919 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:33.740970 28925 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } } }
I20240503 08:18:33.741266 28925 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:33.743536 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.736518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.6.68.253:41951)
0503 08:18:33.736564 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.736566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.736574 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:18:33.736596 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.736597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.736605 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.736659 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.737347 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.737803 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.738384 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.738497 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.740805 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.740811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.740811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.740819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.742473 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.742847 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.742848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.742849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.742869 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.743032 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.743033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.743033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.743159 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.743306 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.743351 (+ 45us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.743439 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20240503 08:18:33.742882 28936 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:18:33.743672 28936 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:18:33.744661 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:33.743217 28851 master_runner.cc:186] Error getting master registration for 127.6.68.254:40951: OK, instance_id { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" instance_seqno: 1714724313415642 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 08:18:33.745226 28851 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 08:18:33.745481 28743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "25ce589605f1406b9bf13c74434c307b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49820
I20240503 08:18:33.756278 28769 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } attempt: 2
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
I20240503 08:18:33.763470 28923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.756708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.6.68.252:43125)
0503 08:18:33.756756 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.756758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.756771 (+ 13us) server_negotiation.cc:407] Connection header received
0503 08:18:33.756857 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.756858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.756869 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.756919 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.757394 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.757835 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.758508 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.758695 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.760745 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.760763 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.760764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.760771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.762407 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.762771 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.762772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.762772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.762790 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.762922 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.762922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.762922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.763069 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.763127 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.763270 (+ 143us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.763365 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240503 08:18:33.763475 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.756667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35284)
0503 08:18:33.756731 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.756743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:33.756789 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:33.756933 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:33.756934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:33.757128 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:33.757330 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.757332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.757915 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.757917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:33.758441 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.758443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.758507 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.759076 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:33.759081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:33.760666 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:33.762494 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.762495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.762496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.762698 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.762844 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.762844 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.762845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.762873 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.763122 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:33.763123 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:33.763172 (+ 49us) client_negotiation.cc:769] Sending connection context
0503 08:18:33.763247 (+ 75us) client_negotiation.cc:240] Negotiation successful
0503 08:18:33.763369 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20240503 08:18:33.765194 28769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b: Bootstrap starting.
I20240503 08:18:33.765775 28769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:33.766013 28769 log.cc:824] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:33.766709 28769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b: No bootstrap required, opened a new log
I20240503 08:18:33.767566 28769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } }
I20240503 08:18:33.767704 28769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ce589605f1406b9bf13c74434c307b, State: Initialized, Role: FOLLOWER
I20240503 08:18:33.767783 28769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } }
I20240503 08:18:33.768030 28769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:33.768105 28743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:18:33.768186 28743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:33.769129 28743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } }
I20240503 08:18:33.769392 28743 leader_election.cc:288] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [CANDIDATE]: Term 1 election: Requested vote from peers 0b43814cd867460990000f0a8300fb1d (127.6.68.253:41951), c2c3a884b80840348d22d8237d527759 (127.6.68.252:43125)
I20240503 08:18:33.771530 28701 master_runner.cc:186] Error getting master registration for 127.6.68.253:41951: OK, instance_id { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" instance_seqno: 1714724313561217 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 08:18:33.773602 28701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 08:18:33.774075 28893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25ce589605f1406b9bf13c74434c307b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c2c3a884b80840348d22d8237d527759"
I20240503 08:18:33.774209 28893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:33.775537 28893 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ce589605f1406b9bf13c74434c307b in term 1.
I20240503 08:18:33.775782 28718 leader_election.cc:302] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ce589605f1406b9bf13c74434c307b, c2c3a884b80840348d22d8237d527759; no voters:
I20240503 08:18:33.776054 28958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [term 1 FOLLOWER]: Leader election won for term 1
W20240503 08:18:33.770510 28950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240503 08:18:33.776343 28958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [term 1 LEADER]: Becoming Leader. State: Replica: 25ce589605f1406b9bf13c74434c307b, State: Running, Role: LEADER
I20240503 08:18:33.776473 28958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } }
W20240503 08:18:33.776899 28950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:18:33.776890 28960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ce589605f1406b9bf13c74434c307b. Latest consensus state: current_term: 1 leader_uuid: "25ce589605f1406b9bf13c74434c307b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } } }
I20240503 08:18:33.777272 28960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:33.778257 28962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:33.778743 28962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:33.781649 28893 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [term 1 FOLLOWER]: Refusing update from remote peer 25ce589605f1406b9bf13c74434c307b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:33.782095 28960 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:33.768669 28939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } } }
I20240503 08:18:33.783228 28939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:33.783499 28925 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ce589605f1406b9bf13c74434c307b. Latest consensus state: current_term: 1 leader_uuid: "25ce589605f1406b9bf13c74434c307b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } } }
I20240503 08:18:33.783584 28925 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:33.786928 28923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.773728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.6.68.252:43125)
0503 08:18:33.773844 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.773846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.773853 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:33.773872 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.773873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.773882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.773932 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.777804 (+ 3872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.778441 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.779135 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.779311 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.780897 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.780901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.780902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.780908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.782791 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.784185 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.784187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.784187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.784219 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.786004 (+ 1785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.786005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.786006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.786129 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.786225 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.786708 (+ 483us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.786837 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20240503 08:18:33.789247 28960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25ce589605f1406b9bf13c74434c307b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } } }
I20240503 08:18:33.789342 28960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:33.789793 28962 catalog_manager.cc:1347] Generated new cluster ID: efaa941021014679b2821619a9906660
I20240503 08:18:33.789844 28962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:33.789825 28925 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25ce589605f1406b9bf13c74434c307b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } } }
I20240503 08:18:33.789902 28925 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:33.790087 28964 mvcc.cc:205] Tried to move back new op lower bound from 7023510789246353408 to 7023510789229072384. Current Snapshot: MvccSnapshot[applied={T|T < 7023510789246353408}]
I20240503 08:18:33.790333 28963 mvcc.cc:205] Tried to move back new op lower bound from 7023510789246353408 to 7023510789229072384. Current Snapshot: MvccSnapshot[applied={T|T < 7023510789246353408 or (T in {7023510789246353408})}]
I20240503 08:18:33.791656 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.770887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.6.68.254:40951)
0503 08:18:33.770930 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.770932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.771437 (+ 505us) server_negotiation.cc:407] Connection header received
0503 08:18:33.771526 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.771527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.771537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.771565 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.772551 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.773192 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.774092 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.774266 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.784157 (+ 9891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.784161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.784161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.784168 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.785872 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.788410 (+ 2538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.788412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.788412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.788440 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.788593 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.788593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.788594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.788701 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.788793 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.791441 (+ 2648us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.791533 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:18:33.791944 28844 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } attempt: 3
May 03 08:18:33 dist-test-slave-3g67 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
I20240503 08:18:33.799008 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.792182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:43125 (local address 127.0.0.1:35302)
0503 08:18:33.792271 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:33.792281 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:33.792319 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:33.792562 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:33.792563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:33.792751 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:33.792939 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.792941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.793554 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.793555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:33.794131 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:33.794133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.794219 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.794724 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:33.794729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:33.796229 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:33.798032 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.798033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.798034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.798288 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.798441 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:33.798442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:33.798443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:33.798471 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:33.798706 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:33.798707 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:33.798749 (+ 42us) client_negotiation.cc:769] Sending connection context
0503 08:18:33.798808 (+ 59us) client_negotiation.cc:240] Negotiation successful
0503 08:18:33.798904 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20240503 08:18:33.799307 28923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.792390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.6.68.252:43125)
0503 08:18:33.792427 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.792429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.792436 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:33.792457 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.792458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.792468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.792513 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.793033 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.793453 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.794218 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.794406 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.796323 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.796327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.796327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.796333 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.797941 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.798364 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.798365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.798365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.798387 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.798526 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.798526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.798527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.798631 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.798683 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.799129 (+ 446us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.799218 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:18:33.800695 28844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d: Bootstrap starting.
I20240503 08:18:33.801371 28844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:33.801635 28844 log.cc:824] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:33.802541 28844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d: No bootstrap required, opened a new log
I20240503 08:18:33.804394 28844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } }
I20240503 08:18:33.804522 28844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b43814cd867460990000f0a8300fb1d, State: Initialized, Role: FOLLOWER
I20240503 08:18:33.804585 28844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } }
I20240503 08:18:33.804903 28844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:33.804927 28818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25ce589605f1406b9bf13c74434c307b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0b43814cd867460990000f0a8300fb1d"
I20240503 08:18:33.805012 28818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:33.805855 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:33.776319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.6.68.253:41951)
0503 08:18:33.776361 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.776362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.787250 (+ 10888us) server_negotiation.cc:407] Connection header received
0503 08:18:33.787352 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.787353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.787366 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.787400 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.788144 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.788815 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:33.795823 (+ 7008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:33.796000 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:33.796041 (+ 41us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:33.796044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:33.796045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:33.796049 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:33.799418 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.805105 (+ 5687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.805107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.805107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.805146 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:33.805306 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:33.805307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:33.805307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:33.805416 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:33.805494 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:33.805681 (+ 187us) server_negotiation.cc:299] Negotiation successful
0503 08:18:33.805762 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:18:33.805958 28818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ce589605f1406b9bf13c74434c307b in term 1.
I20240503 08:18:33.806378 28817 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [term 1 FOLLOWER]: Refusing update from remote peer 25ce589605f1406b9bf13c74434c307b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:33.807092 28939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:33.808967 28982 mvcc.cc:205] Tried to move back new op lower bound from 7023510789246353408 to 7023510789229072384. Current Snapshot: MvccSnapshot[applied={T|T < 7023510789246353408}]
I20240503 08:18:33.812175 28980 catalog_manager.cc:1259] Loaded cluster ID: efaa941021014679b2821619a9906660
I20240503 08:18:33.812276 28980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d: loading cluster ID for follower catalog manager: success
W20240503 08:18:33.812388 28980 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20240503 08:18:33.812448 28980 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:18:33.819748 28969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ce589605f1406b9bf13c74434c307b. Latest consensus state: current_term: 1 leader_uuid: "25ce589605f1406b9bf13c74434c307b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } } }
I20240503 08:18:33.819900 28969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:33.819942 28969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25ce589605f1406b9bf13c74434c307b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } } }
I20240503 08:18:33.819981 28969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:33.819738 28984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25ce589605f1406b9bf13c74434c307b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } } }
I20240503 08:18:33.820076 28984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:33.824774 28939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25ce589605f1406b9bf13c74434c307b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ce589605f1406b9bf13c74434c307b" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40951 } } peers { permanent_uuid: "0b43814cd867460990000f0a8300fb1d" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41951 } } peers { permanent_uuid: "c2c3a884b80840348d22d8237d527759" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 43125 } } }
I20240503 08:18:33.824903 28939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:33.826754 28962 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:33.827564 28962 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:33.836877 28962 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 25ce589605f1406b9bf13c74434c307b: Generated new TSK 0
I20240503 08:18:33.837123 28962 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240503 08:18:33.865605 28923 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:33.864221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.6.68.252:43125)
0503 08:18:33.864267 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.864269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.864855 (+ 586us) server_negotiation.cc:407] Connection header received
0503 08:18:33.864964 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.864966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.864977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.865008 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.865537 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35308: BlockingRecv error: recv got EOF from 127.0.0.1:35308 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20240503 08:18:33.866556 28772 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:33.864360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.6.68.254:40951)
0503 08:18:33.864395 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.864397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.866032 (+ 1635us) server_negotiation.cc:407] Connection header received
0503 08:18:33.866133 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.866134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.866145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.866174 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.866499 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49834: BlockingRecv error: recv got EOF from 127.0.0.1:49834 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20240503 08:18:33.867420 28848 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:33.863985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.6.68.253:41951)
0503 08:18:33.864033 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.864035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.866892 (+ 2857us) server_negotiation.cc:407] Connection header received
0503 08:18:33.867014 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.867015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.867026 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.867055 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.867368 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56142: BlockingRecv error: recv got EOF from 127.0.0.1:56142 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20240503 08:18:33.870206 28848 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:33.869350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.6.68.253:41951)
0503 08:18:33.869388 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.869389 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.869624 (+ 235us) server_negotiation.cc:407] Connection header received
0503 08:18:33.869762 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.869763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.869780 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.869809 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.870157 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56152: BlockingRecv error: recv got EOF from 127.0.0.1:56152 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20240503 08:18:33.873342 28848 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:33.872599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.6.68.253:41951)
0503 08:18:33.872632 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.872634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.872844 (+ 210us) server_negotiation.cc:407] Connection header received
0503 08:18:33.872937 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.872938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.872947 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.872984 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.873295 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56164: BlockingRecv error: recv got EOF from 127.0.0.1:56164 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20240503 08:18:33.877279 28848 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:33.876576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56176 (local address 127.6.68.253:41951)
0503 08:18:33.876606 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.876608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.876803 (+ 195us) server_negotiation.cc:407] Connection header received
0503 08:18:33.876896 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.876897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.876906 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.876935 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.877232 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56176: BlockingRecv error: recv got EOF from 127.0.0.1:56176 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20240503 08:18:33.883365 28848 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:33.882575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.6.68.253:41951)
0503 08:18:33.882621 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.882623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.882849 (+ 226us) server_negotiation.cc:407] Connection header received
0503 08:18:33.882946 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.882947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.882956 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.882984 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.883310 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56186: BlockingRecv error: recv got EOF from 127.0.0.1:56186 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20240503 08:18:33.893785 28848 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:33.892854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.6.68.253:41951)
0503 08:18:33.892894 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.892896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.893138 (+ 242us) server_negotiation.cc:407] Connection header received
0503 08:18:33.893249 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.893250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.893262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.893293 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.893690 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56200: BlockingRecv error: recv got EOF from 127.0.0.1:56200 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20240503 08:18:33.912395 28848 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:33.911429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.6.68.253:41951)
0503 08:18:33.911481 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.911483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.911765 (+ 282us) server_negotiation.cc:407] Connection header received
0503 08:18:33.911874 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.911875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.911887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.911918 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.912335 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56208: BlockingRecv error: recv got EOF from 127.0.0.1:56208 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20240503 08:18:33.946880 28848 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:33.945957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.6.68.253:41951)
0503 08:18:33.945998 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:33.946001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:33.946267 (+ 266us) server_negotiation.cc:407] Connection header received
0503 08:18:33.946376 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:33.946377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:33.946388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:33.946418 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:33.946819 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56218: BlockingRecv error: recv got EOF from 127.0.0.1:56218 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20240503 08:18:34.013171 28848 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:34.012495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.6.68.253:41951)
0503 08:18:34.012541 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:34.012543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:34.012645 (+ 102us) server_negotiation.cc:407] Connection header received
0503 08:18:34.012705 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:34.012706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:34.012718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:34.012776 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:34.013103 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56234: BlockingRecv error: recv got EOF from 127.0.0.1:56234 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20240503 08:18:34.143141 28848 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:34.142398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.6.68.253:41951)
0503 08:18:34.142462 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:34.142465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:34.142562 (+ 97us) server_negotiation.cc:407] Connection header received
0503 08:18:34.142661 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:34.142663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:34.142673 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:34.142722 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:34.143070 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56244: BlockingRecv error: recv got EOF from 127.0.0.1:56244 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20240503 08:18:34.401062 28848 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:34.400404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.6.68.253:41951)
0503 08:18:34.400471 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:34.400473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:34.400485 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:18:34.400543 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:34.400544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:34.400554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:34.400603 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:34.400997 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56248: BlockingRecv error: recv got EOF from 127.0.0.1:56248 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20240503 08:18:34.744086 28936 catalog_manager.cc:1259] Loaded cluster ID: efaa941021014679b2821619a9906660
I20240503 08:18:34.744167 28936 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759: loading cluster ID for follower catalog manager: success
I20240503 08:18:34.745055 28936 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759: acquiring CA information for follower catalog manager: success
I20240503 08:18:34.745347 28936 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2c3a884b80840348d22d8237d527759: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:18:34.813656 28980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d: acquiring CA information for follower catalog manager: success
I20240503 08:18:34.814045 28980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b43814cd867460990000f0a8300fb1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:18:34.917094 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:34.914169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.6.68.253:41951)
0503 08:18:34.914264 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:34.914266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:34.914277 (+ 11us) server_negotiation.cc:407] Connection header received
0503 08:18:34.914358 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:34.914360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:34.914397 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:34.914475 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:34.914768 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:34.915466 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:34.916576 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:34.916748 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:34.916790 (+ 42us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:34.916881 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:34.916949 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:34.917006 (+ 57us) server_negotiation.cc:299] Negotiation successful
0503 08:18:34.917030 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":49,"threads_started":1}
I20240503 08:18:34.920786 29057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:34.918831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.6.68.252:43125)
0503 08:18:34.918933 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:34.918935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:34.918943 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:18:34.918998 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:34.918999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:34.919022 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:34.919090 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:34.919329 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:34.919841 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:34.920388 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:34.920489 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:34.920520 (+ 31us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:34.920602 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:34.920657 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:34.920705 (+ 48us) server_negotiation.cc:299] Negotiation successful
0503 08:18:34.920726 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
I20240503 08:18:34.924737 29057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:34.922538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35324 (local address 127.6.68.252:43125)
0503 08:18:34.922597 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:34.922599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:34.922606 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:34.922628 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:34.922628 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:34.922639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:34.922698 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:34.922921 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:34.923542 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:34.924294 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:34.924434 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:34.924452 (+ 18us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:34.924477 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:34.924532 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:34.924611 (+ 79us) server_negotiation.cc:299] Negotiation successful
0503 08:18:34.924637 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240503 08:18:34.924917 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:34.922450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.6.68.253:41951)
0503 08:18:34.922493 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:34.922494 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:34.922582 (+ 88us) server_negotiation.cc:407] Connection header received
0503 08:18:34.922722 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:34.922723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:34.922737 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:34.922765 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:34.923110 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:34.923744 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:34.924434 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:34.924610 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:34.924630 (+ 20us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:34.924654 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:34.924698 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:34.924801 (+ 103us) server_negotiation.cc:299] Negotiation successful
0503 08:18:34.924833 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240503 08:18:34.925748 29064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:34.922361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.6.68.254:40951)
0503 08:18:34.923158 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:34.923159 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:34.923166 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:34.923211 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:34.923211 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:34.923240 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:34.923321 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:34.923657 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:34.924202 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:34.925191 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:34.925376 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:34.925434 (+ 58us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:34.925506 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:34.925568 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:34.925634 (+ 66us) server_negotiation.cc:299] Negotiation successful
0503 08:18:34.925661 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":54,"threads_started":1}
I20240503 08:18:34.927392 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 28701
I20240503 08:18:34.932972 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 28776
I20240503 08:18:34.938031 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 28851
2024-05-03T08:18:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2720 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 03 08:18:34 dist-test-slave-3g67 krb5kdc[29068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 03 08:18:34 dist-test-slave-3g67 krb5kdc[29068](info): set up 2 sockets
May 03 08:18:34 dist-test-slave-3g67 krb5kdc[29068](info): commencing operation
krb5kdc: starting...
W20240503 08:18:35.893441 6419 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/test-user.keytab.
May 03 08:18:35 dist-test-slave-3g67 krb5kdc[29068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-03T08:18:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:18:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.68.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
Entry for principal kudu/127.6.68.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab.
I20240503 08:18:36.014519 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40357
--webserver_interface=127.6.68.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:44047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40357,127.6.68.253:41485,127.6.68.252:39445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240503 08:18:36.091342 29081 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240503 08:18:36.091545 29081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:36.091562 29081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:36.091583 29081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:36.092900 29081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:36.092940 29081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:36.092954 29081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:36.092965 29081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:36.092979 29081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:36.094486 29081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:44047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40357,127.6.68.253:41485,127.6.68.252:39445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.254.keytab
--principal=kudu/127.6.68.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.254:40357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.68.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:36.094687 29081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:36.094872 29081 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:36.099303 29086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:36.099725 29089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:36.100339 29087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:36.100931 29081 server_base.cc:1034] running on GCE node
I20240503 08:18:36.101150 29081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:36.101409 29081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:36.102555 29081 hybrid_clock.cc:648] HybridClock initialized: now 1714724316102532 us; error 32 us; skew 500 ppm
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:36.105059 29081 init.cc:377] Logged in from keytab as kudu/127.6.68.254@KRBTEST.COM (short username kudu)
I20240503 08:18:36.105450 29081 webserver.cc:460] Webserver started at http://127.6.68.254:33993/ using document root <none> and password file <none>
I20240503 08:18:36.105644 29081 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:36.105691 29081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:36.105800 29081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:36.106820 29081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/data/instance:
uuid: "23e35e72123e43b5af6324b5cf9d7b47"
format_stamp: "Formatted at 2024-05-03 08:18:36 on dist-test-slave-3g67"
server_key: "26fc625cfa78c790c62da1aa084ccb98"
server_key_iv: "f42ac74be4c6ddd504cb845a0294d7b1"
server_key_version: "encryptionkey@0"
I20240503 08:18:36.107133 29081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/wal/instance:
uuid: "23e35e72123e43b5af6324b5cf9d7b47"
format_stamp: "Formatted at 2024-05-03 08:18:36 on dist-test-slave-3g67"
server_key: "26fc625cfa78c790c62da1aa084ccb98"
server_key_iv: "f42ac74be4c6ddd504cb845a0294d7b1"
server_key_version: "encryptionkey@0"
I20240503 08:18:36.108379 29081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:18:36.109025 29096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:36.109187 29081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:36.109262 29081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/wal
uuid: "23e35e72123e43b5af6324b5cf9d7b47"
format_stamp: "Formatted at 2024-05-03 08:18:36 on dist-test-slave-3g67"
server_key: "26fc625cfa78c790c62da1aa084ccb98"
server_key_iv: "f42ac74be4c6ddd504cb845a0294d7b1"
server_key_version: "encryptionkey@0"
I20240503 08:18:36.109319 29081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:36.137220 29081 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:36.139037 29081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:36.139160 29081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:36.142910 29081 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.254:40357
I20240503 08:18:36.142961 29148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.254:40357 every 8 connection(s)
I20240503 08:18:36.143262 29081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/data/info.pb
I20240503 08:18:36.143970 29149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:36.145344 29149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:18:36.148788 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 29081
I20240503 08:18:36.148900 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-0/wal/instance
I20240503 08:18:36.149091 6419 external_mini_cluster.cc:1428] Setting key 0cd64876d052edbaec078b802266e1b2
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:36.154096 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.145926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40357 (local address 127.0.0.1:53934)
0503 08:18:36.146105 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.146134 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:36.146594 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:36.146864 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:36.146866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:36.147229 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:36.147458 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.147462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.148339 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.148340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:36.148751 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.148754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.148904 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.149475 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:36.149496 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:36.151200 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:36.152927 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.152931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.152934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.153099 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.153270 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.153270 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.153271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.153313 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.153600 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:36.153603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:36.153658 (+ 55us) client_negotiation.cc:769] Sending connection context
0503 08:18:36.153738 (+ 80us) client_negotiation.cc:240] Negotiation successful
0503 08:18:36.153895 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20240503 08:18:36.154096 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.145981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.6.68.254:40357)
0503 08:18:36.146111 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.146123 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.146217 (+ 94us) server_negotiation.cc:407] Connection header received
0503 08:18:36.146739 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.146741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.146757 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.146830 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.147550 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.148263 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.148866 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.149073 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.151276 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.151284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.151285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.151292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.152846 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.153170 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.153172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.153173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.153207 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.153367 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.153368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.153368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.153529 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.153617 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.153772 (+ 155us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.153968 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20240503 08:18:36.154768 29149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:18:36.155088 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.154973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41485 (local address 127.0.0.1:46510)
0503 08:18:36.155029 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.155070 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20240503 08:18:36.155169 29099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41485: connect: Connection refused (error 111)
I20240503 08:18:36.155519 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.155438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41485 (local address 127.0.0.1:46520)
0503 08:18:36.155477 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.155498 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240503 08:18:36.155637 29149 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.253:41485: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41485: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.6.68.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab.
Entry for principal kudu/127.6.68.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab.
I20240503 08:18:36.175242 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.253:41485
--webserver_interface=127.6.68.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab
--principal=kudu/127.6.68.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:44047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40357,127.6.68.253:41485,127.6.68.252:39445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:18:36.204818 29149 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } attempt: 1
I20240503 08:18:36.205286 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.205160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41485 (local address 127.0.0.1:46536)
0503 08:18:36.205215 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.205254 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20240503 08:18:36.205606 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.205530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41485 (local address 127.0.0.1:46550)
0503 08:18:36.205565 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.205589 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20240503 08:18:36.205693 29149 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.253:41485: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41485: connect: Connection refused (error 111)
W20240503 08:18:36.256455 29156 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240503 08:18:36.256671 29156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:36.256696 29156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:36.256727 29156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:36.258052 29156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:36.258093 29156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:36.258116 29156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:36.258136 29156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:36.258153 29156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:36.259708 29156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:44047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40357,127.6.68.253:41485,127.6.68.252:39445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.253.keytab
--principal=kudu/127.6.68.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.253:41485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.68.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:36.259930 29156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:36.260113 29156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:36.264529 29164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:36.265151 29162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:36.265630 29161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:36.266283 29156 server_base.cc:1034] running on GCE node
I20240503 08:18:36.266487 29156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:36.266687 29156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:36.267843 29156 hybrid_clock.cc:648] HybridClock initialized: now 1714724316267824 us; error 34 us; skew 500 ppm
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:36.270187 29156 init.cc:377] Logged in from keytab as kudu/127.6.68.253@KRBTEST.COM (short username kudu)
I20240503 08:18:36.270562 29156 webserver.cc:460] Webserver started at http://127.6.68.253:40053/ using document root <none> and password file <none>
I20240503 08:18:36.270737 29156 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:36.270779 29156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:36.270882 29156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:36.271875 29156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/data/instance:
uuid: "cd81add0a23240c9a833bb2b6998c872"
format_stamp: "Formatted at 2024-05-03 08:18:36 on dist-test-slave-3g67"
server_key: "1d5997561d6b3659de0f30b016e56839"
server_key_iv: "5fa90adea1d6daa3efc3dcb3580cacf5"
server_key_version: "encryptionkey@0"
I20240503 08:18:36.272166 29156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/wal/instance:
uuid: "cd81add0a23240c9a833bb2b6998c872"
format_stamp: "Formatted at 2024-05-03 08:18:36 on dist-test-slave-3g67"
server_key: "1d5997561d6b3659de0f30b016e56839"
server_key_iv: "5fa90adea1d6daa3efc3dcb3580cacf5"
server_key_version: "encryptionkey@0"
I20240503 08:18:36.273308 29156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:18:36.273841 29149 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } attempt: 2
I20240503 08:18:36.273964 29171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:36.274161 29156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:36.274214 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.274098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41485 (local address 127.0.0.1:46562)
0503 08:18:36.274147 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.274175 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20240503 08:18:36.274233 29156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/wal
uuid: "cd81add0a23240c9a833bb2b6998c872"
format_stamp: "Formatted at 2024-05-03 08:18:36 on dist-test-slave-3g67"
server_key: "1d5997561d6b3659de0f30b016e56839"
server_key_iv: "5fa90adea1d6daa3efc3dcb3580cacf5"
server_key_version: "encryptionkey@0"
I20240503 08:18:36.274302 29156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:36.274617 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.274496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41485 (local address 127.0.0.1:46568)
0503 08:18:36.274553 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.274588 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20240503 08:18:36.274724 29149 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.253:41485: Network error: Client connection negotiation failed: client connection to 127.6.68.253:41485: connect: Connection refused (error 111)
I20240503 08:18:36.288215 29156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:36.289943 29156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:36.290052 29156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:36.293658 29156 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.253:41485
I20240503 08:18:36.293701 29223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.253:41485 every 8 connection(s)
I20240503 08:18:36.293973 29156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/data/info.pb
I20240503 08:18:36.294785 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:36.296197 29224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:18:36.299933 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 29156
I20240503 08:18:36.300030 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-1/wal/instance
I20240503 08:18:36.300196 6419 external_mini_cluster.cc:1428] Setting key 3773bd7c37411c73f4251a9a3ccf4213
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:36.304507 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.296768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40357 (local address 127.0.0.1:53938)
0503 08:18:36.296921 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.296944 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:36.297386 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:36.297628 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:36.297631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:36.297922 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:36.298185 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.298188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.298807 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.298808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:36.299158 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.299160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.299284 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.299886 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:36.299900 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:36.301447 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:36.303382 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.303391 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.303394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.303645 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.303820 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.303821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.303821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.303870 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.304102 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:36.304105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:36.304201 (+ 96us) client_negotiation.cc:769] Sending connection context
0503 08:18:36.304279 (+ 78us) client_negotiation.cc:240] Negotiation successful
0503 08:18:36.304373 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20240503 08:18:36.304569 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.296803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.6.68.254:40357)
0503 08:18:36.296881 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.296884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.297025 (+ 141us) server_negotiation.cc:407] Connection header received
0503 08:18:36.297491 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.297492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.297505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.297566 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.298289 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.298730 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.299245 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.299439 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.301555 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.301561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.301561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.301570 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.303269 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.303723 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.303724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.303725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.303754 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.303918 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.303919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.303920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.304041 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.304106 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.304330 (+ 224us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.304431 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240503 08:18:36.305109 29224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.68.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.68.252@KRBTEST.COM" created.
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
I20240503 08:18:36.314671 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.305354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41485 (local address 127.0.0.1:46582)
0503 08:18:36.305397 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.305407 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:36.305444 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:36.305680 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:36.305682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:36.305840 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:36.306025 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.306027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.306824 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.306826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:36.307400 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.307403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.307476 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.307937 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:36.307941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:36.311500 (+ 3559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:36.313789 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.313790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.313790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.313967 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.314182 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.314182 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.314182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.314204 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.314453 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:36.314454 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:36.314487 (+ 33us) client_negotiation.cc:769] Sending connection context
0503 08:18:36.314532 (+ 45us) client_negotiation.cc:240] Negotiation successful
0503 08:18:36.314603 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20240503 08:18:36.314816 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.305423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46582 (local address 127.6.68.253:41485)
0503 08:18:36.305559 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.305565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.305574 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:36.305588 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.305590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.305599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.305652 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.306103 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.306730 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.307480 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.307613 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.311607 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.311616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.311617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.311650 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.313703 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.314059 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.314063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.314063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.314105 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.314256 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.314257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.314258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.314397 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.314489 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.314567 (+ 78us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.314679 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20240503 08:18:36.315433 29224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:18:36.315788 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.315675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42318)
0503 08:18:36.315722 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.315767 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240503 08:18:36.315910 29173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
I20240503 08:18:36.316326 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.316246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42324)
0503 08:18:36.316289 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.316311 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240503 08:18:36.316427 29224 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:39445: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.68.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab.
Entry for principal kudu/127.6.68.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab.
I20240503 08:18:36.327579 6419 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
/tmp/dist-test-taskxRG6Mz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.252:39445
--webserver_interface=127.6.68.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab
--principal=kudu/127.6.68.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.68.212:44047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.68.254:40357,127.6.68.253:41485,127.6.68.252:39445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240503 08:18:36.365649 29224 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } attempt: 1
I20240503 08:18:36.365947 29149 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } attempt: 3
I20240503 08:18:36.366199 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.366050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42332)
0503 08:18:36.366120 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.366152 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20240503 08:18:36.366511 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.366435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42336)
0503 08:18:36.366465 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.366484 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20240503 08:18:36.366632 29224 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:39445: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
I20240503 08:18:36.373010 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.366213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41485 (local address 127.0.0.1:46598)
0503 08:18:36.366277 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.366288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:36.366334 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:36.366512 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:36.366513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:36.366720 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:36.366969 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.366972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.367907 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.367908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:36.368278 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.368280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.368342 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.368830 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:36.368835 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:36.370463 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:36.372300 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.372300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.372301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.372433 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.372581 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.372581 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.372582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.372601 (+ 19us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.372790 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:36.372790 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:36.372824 (+ 34us) client_negotiation.cc:769] Sending connection context
0503 08:18:36.372869 (+ 45us) client_negotiation.cc:240] Negotiation successful
0503 08:18:36.372942 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20240503 08:18:36.373055 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.366298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.6.68.253:41485)
0503 08:18:36.366342 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.366344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.366356 (+ 12us) server_negotiation.cc:407] Connection header received
0503 08:18:36.366386 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.366387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.366399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.367051 (+ 652us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.367096 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.367814 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.368356 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.368528 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.370545 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.370549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.370549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.370556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.372211 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.372502 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.372503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.372503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.372527 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.372638 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.372638 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.372638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.372743 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.372800 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.372891 (+ 91us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.372972 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240503 08:18:36.373785 29149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:18:36.374140 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.374032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42350)
0503 08:18:36.374082 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.374115 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20240503 08:18:36.374521 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.374404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42366)
0503 08:18:36.374468 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.374498 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20240503 08:18:36.374655 29149 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:39445: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
I20240503 08:18:36.405800 29149 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } attempt: 1
I20240503 08:18:36.406246 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.406127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42368)
0503 08:18:36.406186 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.406215 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20240503 08:18:36.406625 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.406545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42378)
0503 08:18:36.406589 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.406610 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240503 08:18:36.406703 29149 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:39445: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
W20240503 08:18:36.410763 29231 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240503 08:18:36.410943 29231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:18:36.410971 29231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:18:36.411020 29231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:18:36.412382 29231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:18:36.412425 29231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240503 08:18:36.412447 29231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:18:36.412465 29231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:18:36.412498 29231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:18:36.414016 29231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.68.212:44047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.68.254:40357,127.6.68.253:41485,127.6.68.252:39445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/krb5kdc/kudu_127.6.68.252.keytab
--principal=kudu/127.6.68.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.68.252:39445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.68.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type RELEASE
built by None at 03 May 2024 07:43:15 UTC on efd1337a2aa3
build id 102
I20240503 08:18:36.414227 29231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:18:36.414397 29231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:18:36.418984 29237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:36.419400 29236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:18:36.420039 29239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:18:36.420627 29231 server_base.cc:1034] running on GCE node
I20240503 08:18:36.420847 29231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:18:36.421077 29231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:18:36.422207 29231 hybrid_clock.cc:648] HybridClock initialized: now 1714724316422188 us; error 33 us; skew 500 ppm
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240503 08:18:36.424523 29231 init.cc:377] Logged in from keytab as kudu/127.6.68.252@KRBTEST.COM (short username kudu)
I20240503 08:18:36.424929 29231 webserver.cc:460] Webserver started at http://127.6.68.252:35639/ using document root <none> and password file <none>
I20240503 08:18:36.425112 29231 fs_manager.cc:359] Metadata directory not provided
I20240503 08:18:36.425155 29231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:18:36.425273 29231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:18:36.426239 29231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/data/instance:
uuid: "75c679ac2b324765a2b7331a0fad63d4"
format_stamp: "Formatted at 2024-05-03 08:18:36 on dist-test-slave-3g67"
server_key: "facd86d542ca9090bc2906e711fe49ae"
server_key_iv: "5fe1959d2ea976fd06f1fa9ad81e564a"
server_key_version: "encryptionkey@0"
I20240503 08:18:36.426555 29231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/wal/instance:
uuid: "75c679ac2b324765a2b7331a0fad63d4"
format_stamp: "Formatted at 2024-05-03 08:18:36 on dist-test-slave-3g67"
server_key: "facd86d542ca9090bc2906e711fe49ae"
server_key_iv: "5fe1959d2ea976fd06f1fa9ad81e564a"
server_key_version: "encryptionkey@0"
I20240503 08:18:36.427763 29231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240503 08:18:36.428438 29246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:18:36.428615 29231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:18:36.428690 29231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/data,/tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/wal
uuid: "75c679ac2b324765a2b7331a0fad63d4"
format_stamp: "Formatted at 2024-05-03 08:18:36 on dist-test-slave-3g67"
server_key: "facd86d542ca9090bc2906e711fe49ae"
server_key_iv: "5fe1959d2ea976fd06f1fa9ad81e564a"
server_key_version: "encryptionkey@0"
I20240503 08:18:36.428761 29231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:18:36.434784 29224 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } attempt: 2
I20240503 08:18:36.435214 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.435078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42388)
0503 08:18:36.435136 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.435177 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20240503 08:18:36.435556 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.435479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42400)
0503 08:18:36.435519 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.435540 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240503 08:18:36.435657 29224 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.6.68.252:39445: Network error: Client connection negotiation failed: client connection to 127.6.68.252:39445: connect: Connection refused (error 111)
I20240503 08:18:36.444834 29231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:18:36.446542 29231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:18:36.446652 29231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:18:36.450654 29231 rpc_server.cc:307] RPC server started. Bound to: 127.6.68.252:39445
I20240503 08:18:36.450685 29298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.68.252:39445 every 8 connection(s)
I20240503 08:18:36.451033 29231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/data/info.pb
I20240503 08:18:36.451972 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:18:36.452226 6419 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu as pid 29231
I20240503 08:18:36.452332 6419 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxRG6Mz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714724013808011-6419-0/minicluster-data/master-2/wal/instance
I20240503 08:18:36.452489 6419 external_mini_cluster.cc:1428] Setting key d0e7acff68e0baba96032ccd3bd46384
I20240503 08:18:36.453706 29299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
I20240503 08:18:36.461858 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.454440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.6.68.254:40357)
0503 08:18:36.454491 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.454493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.454590 (+ 97us) server_negotiation.cc:407] Connection header received
0503 08:18:36.454623 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.454624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.454634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.454813 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.455398 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.455982 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.456629 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.456822 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.458639 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.458643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.458643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.458651 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.460251 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.460758 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.460760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.460760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.460789 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.461037 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.461038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.461038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.461130 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.461220 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.461645 (+ 425us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.461759 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240503 08:18:36.462109 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.454318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.254:40357 (local address 127.0.0.1:53942)
0503 08:18:36.454461 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.454491 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:36.454576 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:36.454830 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:36.454832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:36.455110 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:36.455314 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.455318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.456049 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.456050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:36.456548 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.456550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.456636 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.457092 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:36.457106 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:36.458547 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:36.460328 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.460332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.460335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.460659 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.460858 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.460859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.460859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.460953 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.461433 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:36.461437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:36.461514 (+ 77us) client_negotiation.cc:769] Sending connection context
0503 08:18:36.461611 (+ 97us) client_negotiation.cc:240] Negotiation successful
0503 08:18:36.461743 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20240503 08:18:36.462112 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.453006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.6.68.252:39445)
0503 08:18:36.453154 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.453162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.453180 (+ 18us) server_negotiation.cc:407] Connection header received
0503 08:18:36.453587 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.453592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.453631 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.453746 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.454834 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.455679 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.456299 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.456471 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.459067 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.459078 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.459079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.459084 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.460830 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.461162 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.461164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.461165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.461192 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.461355 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.461356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.461356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.461522 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.461641 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.461742 (+ 101us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.461823 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20240503 08:18:36.462623 29299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.254@KRBTEST.COM
I20240503 08:18:36.468938 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.462857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.253:41485 (local address 127.0.0.1:46614)
0503 08:18:36.462916 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.462925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:36.462964 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:36.463092 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:36.463093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:36.463244 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:36.463369 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.463370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.464016 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.464017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:36.464335 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.464336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.464398 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.465087 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:36.465090 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:36.466210 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:36.468026 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.468027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.468027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.468210 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.468369 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.468369 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.468370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.468391 (+ 21us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.468621 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:36.468622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:36.468666 (+ 44us) client_negotiation.cc:769] Sending connection context
0503 08:18:36.468734 (+ 68us) client_negotiation.cc:240] Negotiation successful
0503 08:18:36.468826 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20240503 08:18:36.468941 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.462952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.6.68.253:41485)
0503 08:18:36.462992 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.462994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.463002 (+ 8us) server_negotiation.cc:407] Connection header received
0503 08:18:36.463025 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.463026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.463035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.463090 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.463451 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.463947 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.464404 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.464597 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.466320 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.466324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.466325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.466332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.467947 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.468279 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.468280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.468280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.468302 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.468443 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.468443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.468444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.468557 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.468626 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.468753 (+ 127us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.468830 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240503 08:18:36.469125 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.462874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.6.68.254:40357)
0503 08:18:36.462910 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.462912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.462935 (+ 23us) server_negotiation.cc:407] Connection header received
0503 08:18:36.462954 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.462955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.462963 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.463014 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.463540 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.463989 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.464446 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.464602 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.466818 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.466821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.466822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.466829 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.468146 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.468477 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.468477 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.468478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.468494 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.468632 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.468633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.468633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.468734 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.468879 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.468951 (+ 72us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.469032 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240503 08:18:36.469357 29299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } has no permanent_uuid. Determining permanent_uuid...
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.252@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.68.253@KRBTEST.COM
I20240503 08:18:36.475210 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.469581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42412)
0503 08:18:36.469629 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.469640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:36.469667 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:36.469807 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:36.469808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:36.469947 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:36.470110 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.470113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.470861 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.470862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:36.471184 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.471186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.471254 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.471707 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:36.471710 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:36.472816 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:36.474396 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.474397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.474397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.474565 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.474724 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.474725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.474725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.474765 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.474995 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:36.474996 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:36.475030 (+ 34us) client_negotiation.cc:769] Sending connection context
0503 08:18:36.475089 (+ 59us) client_negotiation.cc:240] Negotiation successful
0503 08:18:36.475151 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20240503 08:18:36.475450 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.469613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.6.68.252:39445)
0503 08:18:36.469647 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.469649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.469676 (+ 27us) server_negotiation.cc:407] Connection header received
0503 08:18:36.469741 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.469742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.469749 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.469791 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.470173 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.470742 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.471258 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.471434 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.472903 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.472906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.472906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.472911 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.474313 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.474640 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.474641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.474641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.474662 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.474829 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.474830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.474831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.474922 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.474987 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.475128 (+ 141us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.475210 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240503 08:18:36.476563 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.470172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.6.68.253:41485)
0503 08:18:36.470215 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.470217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.470224 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:36.470239 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.470239 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.470246 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.470287 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.470787 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.471179 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.471794 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.472232 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.474028 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.474032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.474033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.474039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.475611 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.475938 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.475938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.475939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.475962 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.476083 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.476084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.476084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.476178 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.476281 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.476411 (+ 130us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.476478 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240503 08:18:36.477206 29299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4: Bootstrap starting.
I20240503 08:18:36.477780 29299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:36.477896 6419 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240503 08:18:36.478085 29299 log.cc:824] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:36.478823 29299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4: No bootstrap required, opened a new log
I20240503 08:18:36.478638 29123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "23e35e72123e43b5af6324b5cf9d7b47"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53956
I20240503 08:18:36.480552 29299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } }
I20240503 08:18:36.480683 29299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c679ac2b324765a2b7331a0fad63d4, State: Initialized, Role: FOLLOWER
I20240503 08:18:36.480748 29299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } }
I20240503 08:18:36.481029 29299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:36.481007 29305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } } }
I20240503 08:18:36.481087 29305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:36.481864 29149 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } attempt: 2
W20240503 08:18:36.482789 29316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:18:36.483430 29316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.254@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
I20240503 08:18:36.483924 29231 master_runner.cc:186] Error getting master registration for 127.6.68.254:40357: OK, instance_id { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" instance_seqno: 1714724316141883 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 08:18:36.486804 29231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 08:18:36.488574 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.482193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.6.68.252:39445)
0503 08:18:36.482226 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.482228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.482246 (+ 18us) server_negotiation.cc:407] Connection header received
0503 08:18:36.482307 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.482308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.482317 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.482380 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.482763 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.483261 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.483961 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.484144 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.485958 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.485960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.485961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.485967 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.487579 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.487886 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.487887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.487888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.487909 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.488041 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.488041 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.488042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.488158 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.488219 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.488411 (+ 192us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.488485 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240503 08:18:36.488588 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.482124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42414)
0503 08:18:36.482183 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.482197 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:36.482253 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:36.482404 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:36.482405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:36.482558 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:36.482685 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.482687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.483333 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.483334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:36.483893 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.483896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.483965 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.484454 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:36.484459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:36.485882 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:36.487680 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.487681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.487681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.487824 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.487961 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.487962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.487963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.487989 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.488243 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:36.488244 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:36.488297 (+ 53us) client_negotiation.cc:769] Sending connection context
0503 08:18:36.488387 (+ 90us) client_negotiation.cc:240] Negotiation successful
0503 08:18:36.488488 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20240503 08:18:36.490016 29149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47: Bootstrap starting.
I20240503 08:18:36.490634 29149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:36.490883 29149 log.cc:824] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:36.491557 29149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47: No bootstrap required, opened a new log
I20240503 08:18:36.492413 29149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } }
I20240503 08:18:36.492508 29149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e35e72123e43b5af6324b5cf9d7b47, State: Initialized, Role: FOLLOWER
I20240503 08:18:36.492571 29149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } }
I20240503 08:18:36.492839 29149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:36.492806 29319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } } }
I20240503 08:18:36.492877 29123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:18:36.492990 29123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:36.493945 29123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } }
I20240503 08:18:36.494171 29123 leader_election.cc:288] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [CANDIDATE]: Term 1 election: Requested vote from peers cd81add0a23240c9a833bb2b6998c872 (127.6.68.253:41485), 75c679ac2b324765a2b7331a0fad63d4 (127.6.68.252:39445)
I20240503 08:18:36.494483 29319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:36.495250 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23e35e72123e43b5af6324b5cf9d7b47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "75c679ac2b324765a2b7331a0fad63d4"
I20240503 08:18:36.495662 29273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:18:36.496187 29333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:18:36.496270 29333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:18:36.496565 29273 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23e35e72123e43b5af6324b5cf9d7b47 in term 1.
I20240503 08:18:36.496392 29081 master_runner.cc:186] Error getting master registration for 127.6.68.253:41485: OK, instance_id { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" instance_seqno: 1714724316292651 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240503 08:18:36.498586 29081 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240503 08:18:36.498783 29098 leader_election.cc:302] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23e35e72123e43b5af6324b5cf9d7b47, 75c679ac2b324765a2b7331a0fad63d4; no voters:
I20240503 08:18:36.498907 29319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:18:36.499064 29319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [term 1 LEADER]: Becoming Leader. State: Replica: 23e35e72123e43b5af6324b5cf9d7b47, State: Running, Role: LEADER
I20240503 08:18:36.499159 29319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } }
I20240503 08:18:36.499531 29319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23e35e72123e43b5af6324b5cf9d7b47. Latest consensus state: current_term: 1 leader_uuid: "23e35e72123e43b5af6324b5cf9d7b47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } } }
I20240503 08:18:36.499612 29319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:36.499928 29339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:18:36.500108 29339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:18:36.501719 29273 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [term 1 FOLLOWER]: Refusing update from remote peer 23e35e72123e43b5af6324b5cf9d7b47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:36.502043 29319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:36.504544 29305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23e35e72123e43b5af6324b5cf9d7b47. Latest consensus state: current_term: 1 leader_uuid: "23e35e72123e43b5af6324b5cf9d7b47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } } }
I20240503 08:18:36.504633 29305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:36.506112 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.495589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.6.68.252:39445)
0503 08:18:36.495675 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.495677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.496580 (+ 903us) server_negotiation.cc:407] Connection header received
0503 08:18:36.496721 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.496722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.496743 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.496772 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.498264 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.498911 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.500037 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.500220 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.502471 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.502475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.502475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.502494 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.504243 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.505109 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.505111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.505111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.505135 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.505375 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.505375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.505376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.505494 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.505571 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.505935 (+ 364us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.506023 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240503 08:18:36.513777 29337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23e35e72123e43b5af6324b5cf9d7b47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } } }
I20240503 08:18:36.513900 29337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:36.514355 29305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23e35e72123e43b5af6324b5cf9d7b47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } } }
I20240503 08:18:36.514441 29305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:36.514492 29342 mvcc.cc:205] Tried to move back new op lower bound from 7023510800388370432 to 7023510800381161472. Current Snapshot: MvccSnapshot[applied={T|T < 7023510800388370432}]
I20240503 08:18:36.514866 29341 mvcc.cc:205] Tried to move back new op lower bound from 7023510800388370432 to 7023510800381161472. Current Snapshot: MvccSnapshot[applied={T|T < 7023510800388370432}]
I20240503 08:18:36.517324 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.495838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.6.68.253:41485)
0503 08:18:36.495885 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.495887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.502301 (+ 6414us) server_negotiation.cc:407] Connection header received
0503 08:18:36.502405 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.502407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.502419 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.502449 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.506696 (+ 4247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.507407 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.508246 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.508432 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.512704 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.512709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.512709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.512717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.515977 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.516433 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.516435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.516435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.516462 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.516619 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.516620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.516621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.516733 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.516818 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.517135 (+ 317us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.517224 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240503 08:18:36.517683 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.506865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.6.68.254:40357)
0503 08:18:36.506905 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.506907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.507090 (+ 183us) server_negotiation.cc:407] Connection header received
0503 08:18:36.507229 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.507229 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.507257 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.507287 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.508485 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.509126 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.510147 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.510337 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.512671 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.512675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.512676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.512682 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.515223 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.516690 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.516691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.516692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.516728 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.516985 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.516985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.516986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.517091 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.517169 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.517508 (+ 339us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.517593 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240503 08:18:36.520900 29339 catalog_manager.cc:1347] Generated new cluster ID: 6c75b3a27a9f445dbf106da7c8d9d961
I20240503 08:18:36.520961 29339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:18:36.526798 29224 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } attempt: 3
May 03 08:18:36 dist-test-slave-3g67 krb5kdc[29068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714724316, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.68.253@KRBTEST.COM for kudu/127.6.68.252@KRBTEST.COM
I20240503 08:18:36.534941 29339 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:18:36.535703 29339 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:18:36.536044 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.527098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.68.252:39445 (local address 127.0.0.1:42438)
0503 08:18:36.527537 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0503 08:18:36.527548 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 08:18:36.527593 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:18:36.527862 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 08:18:36.527862 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 08:18:36.528033 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0503 08:18:36.528223 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.528225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.529546 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.529547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0503 08:18:36.530055 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 08:18:36.530056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.530114 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.531011 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0503 08:18:36.531016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0503 08:18:36.532518 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0503 08:18:36.534636 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.534637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.534638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.534818 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.535299 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0503 08:18:36.535299 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0503 08:18:36.535300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0503 08:18:36.535346 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0503 08:18:36.535733 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0503 08:18:36.535734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0503 08:18:36.535794 (+ 60us) client_negotiation.cc:769] Sending connection context
0503 08:18:36.535858 (+ 64us) client_negotiation.cc:240] Negotiation successful
0503 08:18:36.535950 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417}
I20240503 08:18:36.536406 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.527694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.6.68.252:39445)
0503 08:18:36.527731 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.527733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.527740 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:36.527762 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.527763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.527772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.527811 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.528308 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.528949 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.530501 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.530696 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.532621 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0503 08:18:36.532624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0503 08:18:36.532625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0503 08:18:36.532630 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0503 08:18:36.534536 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.534913 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.534914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.534915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.534952 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0503 08:18:36.535419 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0503 08:18:36.535420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0503 08:18:36.535420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0503 08:18:36.535529 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0503 08:18:36.535582 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.536227 (+ 645us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.536321 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240503 08:18:36.537654 29224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872: Bootstrap starting.
I20240503 08:18:36.538280 29224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872: Neither blocks nor log segments found. Creating new log.
I20240503 08:18:36.538523 29224 log.cc:824] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872: Log is configured to *not* fsync() on all Append() calls
I20240503 08:18:36.548344 29224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872: No bootstrap required, opened a new log
I20240503 08:18:36.550068 29224 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } }
I20240503 08:18:36.550200 29224 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd81add0a23240c9a833bb2b6998c872, State: Initialized, Role: FOLLOWER
I20240503 08:18:36.550262 29224 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } }
I20240503 08:18:36.550559 29224 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:18:36.551677 29198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23e35e72123e43b5af6324b5cf9d7b47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cd81add0a23240c9a833bb2b6998c872"
I20240503 08:18:36.551788 29198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:18:36.552760 29198 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23e35e72123e43b5af6324b5cf9d7b47 in term 1.
I20240503 08:18:36.552899 29339 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47: Generated new TSK 0
I20240503 08:18:36.553071 29197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [term 1 FOLLOWER]: Refusing update from remote peer 23e35e72123e43b5af6324b5cf9d7b47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:18:36.553092 29339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:18:36.553339 29319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:18:36.555476 29356 mvcc.cc:205] Tried to move back new op lower bound from 7023510800594362368 to 7023510800381161472. Current Snapshot: MvccSnapshot[applied={T|T < 7023510800388370432}]
I20240503 08:18:36.555616 29355 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23e35e72123e43b5af6324b5cf9d7b47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } } }
I20240503 08:18:36.555755 29355 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:36.555816 29355 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23e35e72123e43b5af6324b5cf9d7b47. Latest consensus state: current_term: 1 leader_uuid: "23e35e72123e43b5af6324b5cf9d7b47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } } }
I20240503 08:18:36.555886 29355 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:36.555923 29355 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23e35e72123e43b5af6324b5cf9d7b47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } } }
I20240503 08:18:36.555963 29355 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:18:36.556993 29365 catalog_manager.cc:1259] Loaded cluster ID: 6c75b3a27a9f445dbf106da7c8d9d961
I20240503 08:18:36.557046 29365 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872: loading cluster ID for follower catalog manager: success
I20240503 08:18:36.558282 29365 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872: acquiring CA information for follower catalog manager: success
I20240503 08:18:36.558692 29365 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd81add0a23240c9a833bb2b6998c872: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240503 08:18:36.583354 29152 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.581670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.6.68.254:40357)
0503 08:18:36.581717 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.581720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.582678 (+ 958us) server_negotiation.cc:407] Connection header received
0503 08:18:36.582796 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.582797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.582808 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.582840 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.583285 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53966: BlockingRecv error: recv got EOF from 127.0.0.1:53966 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20240503 08:18:36.584370 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.582350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.6.68.252:39445)
0503 08:18:36.582388 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.582390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.583700 (+ 1310us) server_negotiation.cc:407] Connection header received
0503 08:18:36.583861 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.583862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.583874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.583902 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.584316 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42454: BlockingRecv error: recv got EOF from 127.0.0.1:42454 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20240503 08:18:36.584841 29228 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.582053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.6.68.253:41485)
0503 08:18:36.582092 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.582094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.583614 (+ 1520us) server_negotiation.cc:407] Connection header received
0503 08:18:36.583970 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.583971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.583982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.584010 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.584785 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46638: BlockingRecv error: recv got EOF from 127.0.0.1:46638 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20240503 08:18:36.594776 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.587916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46650 (local address 127.6.68.253:41485)
0503 08:18:36.587949 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.587950 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.591809 (+ 3859us) server_negotiation.cc:407] Connection header received
0503 08:18:36.591928 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.591929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.591958 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.592008 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:36.592411 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.593172 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.594093 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.594262 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.594389 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:36.594503 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:36.594551 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.594664 (+ 113us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.594687 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20240503 08:18:36.598075 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.597256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42468 (local address 127.6.68.252:39445)
0503 08:18:36.597293 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.597295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.597519 (+ 224us) server_negotiation.cc:407] Connection header received
0503 08:18:36.597626 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.597626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.597641 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.597670 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.598022 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42468: BlockingRecv error: recv got EOF from 127.0.0.1:42468 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:18:36.603053 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.600255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46664 (local address 127.6.68.253:41485)
0503 08:18:36.600286 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.600288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.600510 (+ 222us) server_negotiation.cc:407] Connection header received
0503 08:18:36.600600 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.600600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.600611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.600640 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:36.600987 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.601732 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.602528 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.602687 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.602800 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:36.602828 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:36.602855 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.602960 (+ 105us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.602984 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20240503 08:18:36.606091 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.605341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42470 (local address 127.6.68.252:39445)
0503 08:18:36.605376 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.605378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.605569 (+ 191us) server_negotiation.cc:407] Connection header received
0503 08:18:36.605662 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.605663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.605673 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.605700 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.606039 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42470: BlockingRecv error: recv got EOF from 127.0.0.1:42470 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20240503 08:18:36.612035 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.609172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.6.68.253:41485)
0503 08:18:36.609205 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.609206 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.609430 (+ 224us) server_negotiation.cc:407] Connection header received
0503 08:18:36.609519 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.609519 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.609529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.609555 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:36.609910 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.610627 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.611465 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.611655 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.611791 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:36.611820 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:36.611847 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.611946 (+ 99us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.611965 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20240503 08:18:36.615012 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.614261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.6.68.252:39445)
0503 08:18:36.614295 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.614297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.614465 (+ 168us) server_negotiation.cc:407] Connection header received
0503 08:18:36.614559 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.614560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.614571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.614599 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.614960 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42484: BlockingRecv error: recv got EOF from 127.0.0.1:42484 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20240503 08:18:36.622997 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.620162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.6.68.253:41485)
0503 08:18:36.620200 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.620202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.620429 (+ 227us) server_negotiation.cc:407] Connection header received
0503 08:18:36.620521 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.620522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.620533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.620561 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:36.620924 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.621682 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.622497 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.622660 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.622772 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:36.622799 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:36.622824 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.622913 (+ 89us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.622931 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20240503 08:18:36.625989 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.625232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.6.68.252:39445)
0503 08:18:36.625271 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.625273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.625463 (+ 190us) server_negotiation.cc:407] Connection header received
0503 08:18:36.625557 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.625558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.625568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.625596 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.625938 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42496: BlockingRecv error: recv got EOF from 127.0.0.1:42496 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20240503 08:18:36.638160 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.635189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.6.68.253:41485)
0503 08:18:36.635230 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.635232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.635486 (+ 254us) server_negotiation.cc:407] Connection header received
0503 08:18:36.635586 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.635587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.635599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.635648 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:36.636048 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.636804 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.637626 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.637805 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.637912 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:36.637939 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:36.637965 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.638064 (+ 99us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.638083 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20240503 08:18:36.641486 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.640686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.6.68.252:39445)
0503 08:18:36.640722 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.640724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.640918 (+ 194us) server_negotiation.cc:407] Connection header received
0503 08:18:36.641012 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.641013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.641024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.641052 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.641430 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42508: BlockingRecv error: recv got EOF from 127.0.0.1:42508 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20240503 08:18:36.661844 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.658841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.6.68.253:41485)
0503 08:18:36.658887 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.658889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.659171 (+ 282us) server_negotiation.cc:407] Connection header received
0503 08:18:36.659292 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.659293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.659304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.659335 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:36.659764 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.660491 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.661269 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.661464 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.661588 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:36.661616 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:36.661643 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.661744 (+ 101us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.661767 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20240503 08:18:36.665019 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.664199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42510 (local address 127.6.68.252:39445)
0503 08:18:36.664242 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.664244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.664456 (+ 212us) server_negotiation.cc:407] Connection header received
0503 08:18:36.664559 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.664560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.664572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.664602 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.664963 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42510: BlockingRecv error: recv got EOF from 127.0.0.1:42510 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:18:36.701642 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.698489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.6.68.253:41485)
0503 08:18:36.698548 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.698550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.698881 (+ 331us) server_negotiation.cc:407] Connection header received
0503 08:18:36.698987 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.698988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.698999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.699031 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:36.699428 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.700181 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.701055 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.701225 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.701371 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:36.701403 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:36.701431 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.701535 (+ 104us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.701558 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20240503 08:18:36.704901 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.704055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.6.68.252:39445)
0503 08:18:36.704099 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.704102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.704335 (+ 233us) server_negotiation.cc:407] Connection header received
0503 08:18:36.704431 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.704432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.704444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.704472 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.704846 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42518: BlockingRecv error: recv got EOF from 127.0.0.1:42518 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20240503 08:18:36.772096 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.770268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.6.68.253:41485)
0503 08:18:36.770366 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.770369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.770421 (+ 52us) server_negotiation.cc:407] Connection header received
0503 08:18:36.770483 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.770484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.770493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.770535 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:36.770767 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.771234 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.771759 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.771858 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.771873 (+ 15us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:36.771892 (+ 19us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:36.771932 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.771991 (+ 59us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.772006 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20240503 08:18:36.774513 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.773951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.6.68.252:39445)
0503 08:18:36.774026 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.774028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.774037 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:36.774084 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.774085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.774096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.774150 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.774454 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42532: BlockingRecv error: recv got EOF from 127.0.0.1:42532 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20240503 08:18:36.905553 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:36.903698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46718 (local address 127.6.68.253:41485)
0503 08:18:36.903770 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.903772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.903785 (+ 13us) server_negotiation.cc:407] Connection header received
0503 08:18:36.903832 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.903833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.903843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.903908 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:36.904140 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.904628 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:36.905176 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:36.905301 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:36.905329 (+ 28us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:36.905353 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:36.905414 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:36.905476 (+ 62us) server_negotiation.cc:299] Negotiation successful
0503 08:18:36.905493 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20240503 08:18:36.907687 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:36.907131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.6.68.252:39445)
0503 08:18:36.907199 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:36.907200 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:36.907210 (+ 10us) server_negotiation.cc:407] Connection header received
0503 08:18:36.907278 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:36.907279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:36.907289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:36.907348 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:36.907596 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42546: BlockingRecv error: recv got EOF from 127.0.0.1:42546 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20240503 08:18:37.166862 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:37.164919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46730 (local address 127.6.68.253:41485)
0503 08:18:37.164988 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:37.164990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:37.165033 (+ 43us) server_negotiation.cc:407] Connection header received
0503 08:18:37.165111 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:37.165111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:37.165121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:37.165160 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:37.165439 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.165920 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:37.166466 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.166587 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:37.166604 (+ 17us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:37.166626 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:37.166674 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:37.166736 (+ 62us) server_negotiation.cc:299] Negotiation successful
0503 08:18:37.166752 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20240503 08:18:37.169070 29302 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:18:37.168547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.6.68.252:39445)
0503 08:18:37.168612 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:37.168614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:37.168623 (+ 9us) server_negotiation.cc:407] Connection header received
0503 08:18:37.168652 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:37.168653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:37.168663 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:37.168710 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0503 08:18:37.169008 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42548: BlockingRecv error: recv got EOF from 127.0.0.1:42548 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20240503 08:18:37.176456 29466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23e35e72123e43b5af6324b5cf9d7b47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e35e72123e43b5af6324b5cf9d7b47" member_type: VOTER last_known_addr { host: "127.6.68.254" port: 40357 } } peers { permanent_uuid: "cd81add0a23240c9a833bb2b6998c872" member_type: VOTER last_known_addr { host: "127.6.68.253" port: 41485 } } peers { permanent_uuid: "75c679ac2b324765a2b7331a0fad63d4" member_type: VOTER last_known_addr { host: "127.6.68.252" port: 39445 } } }
I20240503 08:18:37.176571 29466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e35e72123e43b5af6324b5cf9d7b47 [sys.catalog]: This master's current role is: LEADER
I20240503 08:18:37.483850 29316 catalog_manager.cc:1259] Loaded cluster ID: 6c75b3a27a9f445dbf106da7c8d9d961
I20240503 08:18:37.483928 29316 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4: loading cluster ID for follower catalog manager: success
I20240503 08:18:37.484764 29316 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4: acquiring CA information for follower catalog manager: success
I20240503 08:18:37.485006 29316 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 75c679ac2b324765a2b7331a0fad63d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:18:37.684417 29484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:37.682296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.6.68.253:41485)
0503 08:18:37.682417 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:37.682420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:37.682471 (+ 51us) server_negotiation.cc:407] Connection header received
0503 08:18:37.682558 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:37.682559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:37.682583 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:37.682639 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:37.682931 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.683442 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:37.684019 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.684147 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:37.684162 (+ 15us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:37.684184 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:37.684232 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:37.684306 (+ 74us) server_negotiation.cc:299] Negotiation successful
0503 08:18:37.684333 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20240503 08:18:37.687587 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:37.685825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.6.68.252:39445)
0503 08:18:37.685910 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:37.685911 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:37.685918 (+ 7us) server_negotiation.cc:407] Connection header received
0503 08:18:37.685939 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:37.685940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:37.685976 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:37.686044 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:37.686237 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.686683 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:37.687143 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.687243 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:37.687277 (+ 34us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:37.687361 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:37.687419 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:37.687470 (+ 51us) server_negotiation.cc:299] Negotiation successful
0503 08:18:37.687493 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":43,"threads_started":1}
I20240503 08:18:37.691385 29484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:37.689099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.6.68.253:41485)
0503 08:18:37.689174 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:37.689176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:37.689454 (+ 278us) server_negotiation.cc:407] Connection header received
0503 08:18:37.689479 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:37.689479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:37.689493 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:37.689548 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:37.689807 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.690296 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:37.690867 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.691021 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:37.691037 (+ 16us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:37.691060 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:37.691104 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:37.691297 (+ 193us) server_negotiation.cc:299] Negotiation successful
0503 08:18:37.691317 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240503 08:18:37.691447 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:37.689123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42568 (local address 127.6.68.252:39445)
0503 08:18:37.689203 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:37.689205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:37.689267 (+ 62us) server_negotiation.cc:407] Connection header received
0503 08:18:37.689319 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:37.689320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:37.689331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:37.689376 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:37.689666 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.690303 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:37.691037 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.691185 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:37.691200 (+ 15us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:37.691218 (+ 18us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:37.691266 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:37.691336 (+ 70us) server_negotiation.cc:299] Negotiation successful
0503 08:18:37.691359 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20240503 08:18:37.692171 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0503 08:18:37.689298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.6.68.254:40357)
0503 08:18:37.689399 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0503 08:18:37.689401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0503 08:18:37.689445 (+ 44us) server_negotiation.cc:407] Connection header received
0503 08:18:37.689526 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0503 08:18:37.689526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0503 08:18:37.689554 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0503 08:18:37.689630 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0503 08:18:37.689941 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.690700 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0503 08:18:37.691501 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0503 08:18:37.691803 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0503 08:18:37.691859 (+ 56us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0503 08:18:37.691924 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0503 08:18:37.691984 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0503 08:18:37.692056 (+ 72us) server_negotiation.cc:299] Negotiation successful
0503 08:18:37.692078 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":51,"threads_started":1}
I20240503 08:18:37.693796 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 29081
I20240503 08:18:37.699195 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 29156
I20240503 08:18:37.704061 6419 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxRG6Mz/build/release/bin/kudu with pid 29231
2024-05-03T08:18:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2765 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5486 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (303910 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
1 FAILED TEST